2022-12-01T10:00:26.9885206Z Requested labels: linux.2xlarge
2022-12-01T10:00:26.9885343Z Job defined at: pytorch/pytorch/.github/workflows/_linux-test.yml@refs/pull/89997/merge
2022-12-01T10:00:26.9885366Z Waiting for a runner to pick up this job...
2022-12-01T10:00:27.6068059Z Job is about to start running on the runner: i-0ede55ee7772c82ca (organization)
2022-12-01T10:00:33.0380975Z Current runner version: '2.299.1'
2022-12-01T10:00:33.0386615Z Runner name: 'i-0ede55ee7772c82ca'
2022-12-01T10:00:33.0387159Z Runner group name: 'Default'
2022-12-01T10:00:33.0387836Z Machine name: 'ip-10-0-5-106'
2022-12-01T10:00:33.0389823Z ##[group]GITHUB_TOKEN Permissions
2022-12-01T10:00:33.0390538Z Actions: read
2022-12-01T10:00:33.0390861Z Checks: read
2022-12-01T10:00:33.0391181Z Contents: read
2022-12-01T10:00:33.0391462Z Deployments: read
2022-12-01T10:00:33.0391852Z Discussions: read
2022-12-01T10:00:33.0392175Z Issues: read
2022-12-01T10:00:33.0392451Z Metadata: read
2022-12-01T10:00:33.0392770Z Packages: read
2022-12-01T10:00:33.0393087Z Pages: read
2022-12-01T10:00:33.0393433Z PullRequests: read
2022-12-01T10:00:33.0393762Z RepositoryProjects: read
2022-12-01T10:00:33.0394113Z SecurityEvents: read
2022-12-01T10:00:33.0394441Z Statuses: read
2022-12-01T10:00:33.0394713Z ##[endgroup]
2022-12-01T10:00:33.0397877Z Secret source: None
2022-12-01T10:00:33.0398538Z Prepare workflow directory
2022-12-01T10:00:33.6874864Z Prepare all required actions
2022-12-01T10:00:33.7054173Z Getting action download info
2022-12-01T10:00:33.9613611Z Download action repository 'pytorch/pytorch@master' (SHA:850b53bbee82fb194af85b566aedee94b96def32)
2022-12-01T10:00:36.7805312Z Download action repository 'pytorch/test-infra@main' (SHA:1f415583bdcd967e33ea8fd05be71ed0bdf19880)
2022-12-01T10:00:37.1341637Z Download action repository 'nick-fields/retry@7d4a37704547a311dbb66ebdf5b23ec19374a767' (SHA:7d4a37704547a311dbb66ebdf5b23ec19374a767)
2022-12-01T10:00:37.2519806Z Download action repository 'seemethere/upload-artifact-s3@v5' (SHA:baba72d0712b404f646cebe0730933554ebce96a)
2022-12-01T10:00:37.4999495Z Getting action download info
2022-12-01T10:00:37.6498160Z Download action repository 'malfet/checkout@silent-checkout' (SHA:c7b8fef48edfe1bca0044a44b1f7f7c4318a3076)
2022-12-01T10:00:37.8165653Z Uses: pytorch/pytorch/.github/workflows/_linux-test.yml
2022-12-01T10:00:37.8167333Z ##[group] Inputs
2022-12-01T10:00:37.8167665Z   build-environment: linux-focal-py3.7-gcc7
2022-12-01T10:00:37.8168482Z   test-matrix: { include: [
  { config: "default", shard: 1, num_shards: 2, runner: "linux.2xlarge" },
  { config: "default", shard: 2, num_shards: 2, runner: "linux.2xlarge" },
  { config: "distributed", shard: 1, num_shards: 2, runner: "linux.2xlarge" },
  { config: "distributed", shard: 2, num_shards: 2, runner: "linux.2xlarge" },
  { config: "functorch", shard: 1, num_shards: 1, runner: "linux.2xlarge" },
  { config: "docs_test", shard: 1, num_shards: 1,  runner: "linux.2xlarge" },
  { config: "jit_legacy", shard: 1, num_shards: 1, runner: "linux.2xlarge" },
  { config: "backwards_compat", shard: 1, num_shards: 1, runner: "linux.2xlarge" },
]}

2022-12-01T10:00:37.8169377Z   docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-py3.7-gcc7:fa72f5a0a230eb632055220542038bd4ceca184b
2022-12-01T10:00:37.8169740Z   sync-tag: 
2022-12-01T10:00:37.8169966Z ##[endgroup]
2022-12-01T10:00:37.8170666Z Complete job name: linux-focal-py3.7-gcc7 / test (distributed, 2, 2, linux.2xlarge)
2022-12-01T10:00:37.8925055Z ##[group]Run pytorch/pytorch/.github/actions/checkout-pytorch@master
2022-12-01T10:00:37.8925387Z with:
2022-12-01T10:00:37.8925592Z   submodules: recursive
2022-12-01T10:00:37.8925813Z   fetch-depth: 0
2022-12-01T10:00:37.8926015Z env:
2022-12-01T10:00:37.8926214Z   GIT_DEFAULT_BRANCH: master
2022-12-01T10:00:37.8926439Z ##[endgroup]
2022-12-01T10:00:37.9146269Z ##[group]Run retry () {
2022-12-01T10:00:37.9146575Z retry () {
2022-12-01T10:00:37.9146830Z   $* || (sleep 1 && $*) || (sleep 2 && $*) || (sleep 4 && $*) || (sleep 8 && $*)
2022-12-01T10:00:37.9147081Z }
2022-12-01T10:00:37.9147305Z echo "${GITHUB_WORKSPACE}"
2022-12-01T10:00:37.9147546Z if [ -z "${NO_SUDO}" ]; then
2022-12-01T10:00:37.9147825Z   retry sudo rm -rf "${GITHUB_WORKSPACE}"
2022-12-01T10:00:37.9148104Z else
2022-12-01T10:00:37.9148329Z   retry rm -rf "${GITHUB_WORKSPACE}"
2022-12-01T10:00:37.9148714Z fi
2022-12-01T10:00:37.9148979Z mkdir "${GITHUB_WORKSPACE}"
2022-12-01T10:00:37.9164713Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-12-01T10:00:37.9164979Z env:
2022-12-01T10:00:37.9165197Z   GIT_DEFAULT_BRANCH: master
2022-12-01T10:00:37.9165418Z   NO_SUDO: 
2022-12-01T10:00:37.9165616Z ##[endgroup]
2022-12-01T10:00:37.9334008Z /home/ec2-user/actions-runner/_work/pytorch/pytorch
2022-12-01T10:00:41.5175669Z ##[group]Run malfet/checkout@silent-checkout
2022-12-01T10:00:41.5175923Z with:
2022-12-01T10:00:41.5176123Z   ref: c13d400bffe90e16b96520bbc8a41a6f0c9cd584
2022-12-01T10:00:41.5176321Z   fetch-depth: 0
2022-12-01T10:00:41.5176502Z   submodules: recursive
2022-12-01T10:00:41.5176689Z   quiet-checkout: true
2022-12-01T10:00:41.5176876Z   repository: pytorch/pytorch
2022-12-01T10:00:41.5177255Z   token: ***
2022-12-01T10:00:41.5177429Z   ssh-strict: true
2022-12-01T10:00:41.5177622Z   persist-credentials: true
2022-12-01T10:00:41.5177808Z   clean: true
2022-12-01T10:00:41.5177979Z   lfs: false
2022-12-01T10:00:41.5178161Z   set-safe-directory: true
2022-12-01T10:00:41.5178327Z env:
2022-12-01T10:00:41.5178497Z   GIT_DEFAULT_BRANCH: master
2022-12-01T10:00:41.5178682Z ##[endgroup]
2022-12-01T10:00:41.6351025Z Syncing repository: pytorch/pytorch
2022-12-01T10:00:41.6352514Z ##[group]Getting Git version info
2022-12-01T10:00:41.6352987Z Working directory is '/home/ec2-user/actions-runner/_work/pytorch/pytorch'
2022-12-01T10:00:41.6353432Z [command]/usr/bin/git version
2022-12-01T10:00:41.6353626Z git version 2.37.1
2022-12-01T10:00:41.6354215Z ##[endgroup]
2022-12-01T10:00:41.6365623Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/17d5464f-1063-4a37-8fa9-227c2dc82a21' before making global git config changes
2022-12-01T10:00:41.6366043Z Adding repository directory to the temporary git global config as a safe directory
2022-12-01T10:00:41.6369987Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch
2022-12-01T10:00:41.6411212Z Deleting the contents of '/home/ec2-user/actions-runner/_work/pytorch/pytorch'
2022-12-01T10:00:41.6415431Z ##[group]Initializing the repository
2022-12-01T10:00:41.6417840Z [command]/usr/bin/git init /home/ec2-user/actions-runner/_work/pytorch/pytorch
2022-12-01T10:00:41.6519217Z hint: Using 'master' as the name for the initial branch. This default branch name
2022-12-01T10:00:41.6519697Z hint: is subject to change. To configure the initial branch name to use in all
2022-12-01T10:00:41.6520239Z hint: of your new repositories, which will suppress this warning, call:
2022-12-01T10:00:41.6520650Z hint: 
2022-12-01T10:00:41.6520985Z hint: 	git config --global init.defaultBranch <name>
2022-12-01T10:00:41.6521396Z hint: 
2022-12-01T10:00:41.6521689Z hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and
2022-12-01T10:00:41.6522049Z hint: 'development'. The just-created branch can be renamed via this command:
2022-12-01T10:00:41.6522298Z hint: 
2022-12-01T10:00:41.6522866Z hint: 	git branch -m <name>
2022-12-01T10:00:41.6523234Z Initialized empty Git repository in /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/
2022-12-01T10:00:41.6530908Z [command]/usr/bin/git remote add origin https://github.com/pytorch/pytorch
2022-12-01T10:00:41.6564960Z ##[endgroup]
2022-12-01T10:00:41.6565330Z ##[group]Disabling automatic garbage collection
2022-12-01T10:00:41.6568205Z [command]/usr/bin/git config --local gc.auto 0
2022-12-01T10:00:41.6595506Z ##[endgroup]
2022-12-01T10:00:41.6596610Z ##[group]Setting up auth
2022-12-01T10:00:41.6602240Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand
2022-12-01T10:00:41.6632812Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :
2022-12-01T10:00:41.6875842Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader
2022-12-01T10:00:41.6908800Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :
2022-12-01T10:00:41.7151669Z [command]/usr/bin/git config --local http.https://github.com/.extraheader AUTHORIZATION: basic ***
2022-12-01T10:00:41.7200571Z ##[endgroup]
2022-12-01T10:00:41.7200946Z ##[group]Fetching the repository
2022-12-01T10:00:41.7207125Z [command]/usr/bin/git -c protocol.version=2 fetch --prune --quiet --no-recurse-submodules origin +refs/heads/*:refs/remotes/origin/* +refs/tags/*:refs/tags/*
2022-12-01T10:01:35.6519851Z [command]/usr/bin/git rev-parse --verify --quiet c13d400bffe90e16b96520bbc8a41a6f0c9cd584^{object}
2022-12-01T10:01:35.6554862Z [command]/usr/bin/git -c protocol.version=2 fetch --no-tags --prune --quiet --no-recurse-submodules origin c13d400bffe90e16b96520bbc8a41a6f0c9cd584
2022-12-01T10:01:36.6583319Z ##[endgroup]
2022-12-01T10:01:36.6583860Z ##[group]Determining the checkout info
2022-12-01T10:01:36.6584481Z ##[endgroup]
2022-12-01T10:01:36.6584977Z ##[group]Checking out the ref
2022-12-01T10:01:36.6588144Z [command]/usr/bin/git checkout --quiet --force c13d400bffe90e16b96520bbc8a41a6f0c9cd584
2022-12-01T10:01:37.8909944Z ##[endgroup]
2022-12-01T10:01:37.8910409Z ##[group]Setting up auth for fetching submodules
2022-12-01T10:01:37.8915252Z [command]/usr/bin/git config --global http.https://github.com/.extraheader AUTHORIZATION: basic ***
2022-12-01T10:01:37.8961914Z [command]/usr/bin/git config --global --unset-all url.https://github.com/.insteadOf
2022-12-01T10:01:37.8992057Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf git@github.com:
2022-12-01T10:01:37.9019509Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf org-21003710@github.com:
2022-12-01T10:01:37.9043521Z ##[endgroup]
2022-12-01T10:01:37.9043932Z ##[group]Fetching submodules
2022-12-01T10:01:37.9047432Z [command]/usr/bin/git submodule sync --recursive
2022-12-01T10:01:37.9309208Z [command]/usr/bin/git -c protocol.version=2 submodule update --init --force --recursive
2022-12-01T10:01:37.9564690Z Submodule 'android/libs/fbjni' (https://github.com/facebookincubator/fbjni.git) registered for path 'android/libs/fbjni'
2022-12-01T10:01:37.9567228Z Submodule 'third_party/NNPACK_deps/FP16' (https://github.com/Maratyszcza/FP16.git) registered for path 'third_party/FP16'
2022-12-01T10:01:37.9567880Z Submodule 'third_party/NNPACK_deps/FXdiv' (https://github.com/Maratyszcza/FXdiv.git) registered for path 'third_party/FXdiv'
2022-12-01T10:01:37.9570435Z Submodule 'third_party/NNPACK' (https://github.com/Maratyszcza/NNPACK.git) registered for path 'third_party/NNPACK'
2022-12-01T10:01:37.9573034Z Submodule 'third_party/QNNPACK' (https://github.com/pytorch/QNNPACK) registered for path 'third_party/QNNPACK'
2022-12-01T10:01:37.9576062Z Submodule 'third_party/VulkanMemoryAllocator' (https://github.com/GPUOpen-LibrariesAndSDKs/VulkanMemoryAllocator.git) registered for path 'third_party/VulkanMemoryAllocator'
2022-12-01T10:01:37.9578586Z Submodule 'third_party/XNNPACK' (https://github.com/google/XNNPACK.git) registered for path 'third_party/XNNPACK'
2022-12-01T10:01:37.9582804Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/benchmark'
2022-12-01T10:01:37.9585738Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo.git) registered for path 'third_party/cpuinfo'
2022-12-01T10:01:37.9588889Z Submodule 'third_party/cub' (https://github.com/NVlabs/cub.git) registered for path 'third_party/cub'
2022-12-01T10:01:37.9592624Z Submodule 'third_party/cudnn_frontend' (https://github.com/NVIDIA/cudnn-frontend.git) registered for path 'third_party/cudnn_frontend'
2022-12-01T10:01:37.9595918Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/cutlass'
2022-12-01T10:01:37.9599279Z Submodule 'third_party/eigen' (https://gitlab.com/libeigen/eigen.git) registered for path 'third_party/eigen'
2022-12-01T10:01:37.9602771Z Submodule 'third_party/fbgemm' (https://github.com/pytorch/fbgemm) registered for path 'third_party/fbgemm'
2022-12-01T10:01:37.9606526Z Submodule 'third_party/flatbuffers' (https://github.com/google/flatbuffers.git) registered for path 'third_party/flatbuffers'
2022-12-01T10:01:37.9610149Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/fmt'
2022-12-01T10:01:37.9613904Z Submodule 'third_party/foxi' (https://github.com/houseroad/foxi.git) registered for path 'third_party/foxi'
2022-12-01T10:01:37.9617865Z Submodule 'third_party/gemmlowp/gemmlowp' (https://github.com/google/gemmlowp.git) registered for path 'third_party/gemmlowp/gemmlowp'
2022-12-01T10:01:37.9622165Z Submodule 'third_party/gloo' (https://github.com/facebookincubator/gloo) registered for path 'third_party/gloo'
2022-12-01T10:01:37.9626215Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/googletest'
2022-12-01T10:01:37.9630382Z Submodule 'third_party/ideep' (https://github.com/intel/ideep) registered for path 'third_party/ideep'
2022-12-01T10:01:37.9634863Z Submodule 'third_party/ios-cmake' (https://github.com/Yangqing/ios-cmake.git) registered for path 'third_party/ios-cmake'
2022-12-01T10:01:37.9639252Z Submodule 'third_party/ittapi' (https://github.com/intel/ittapi.git) registered for path 'third_party/ittapi'
2022-12-01T10:01:37.9643962Z Submodule 'third_party/kineto' (https://github.com/pytorch/kineto) registered for path 'third_party/kineto'
2022-12-01T10:01:37.9648502Z Submodule 'third_party/nccl/nccl' (https://github.com/NVIDIA/nccl) registered for path 'third_party/nccl/nccl'
2022-12-01T10:01:37.9653306Z Submodule 'third_party/neon2sse' (https://github.com/intel/ARM_NEON_2_x86_SSE.git) registered for path 'third_party/neon2sse'
2022-12-01T10:01:37.9658024Z Submodule 'third_party/nlohmann' (https://github.com/nlohmann/json.git) registered for path 'third_party/nlohmann'
2022-12-01T10:01:37.9663687Z Submodule 'third_party/onnx' (https://github.com/onnx/onnx.git) registered for path 'third_party/onnx'
2022-12-01T10:01:37.9668687Z Submodule 'third_party/onnx-tensorrt' (https://github.com/onnx/onnx-tensorrt) registered for path 'third_party/onnx-tensorrt'
2022-12-01T10:01:37.9673783Z Submodule 'third_party/pocketfft' (https://github.com/mreineck/pocketfft) registered for path 'third_party/pocketfft'
2022-12-01T10:01:37.9678931Z Submodule 'third_party/protobuf' (https://github.com/protocolbuffers/protobuf.git) registered for path 'third_party/protobuf'
2022-12-01T10:01:37.9684226Z Submodule 'third_party/NNPACK_deps/psimd' (https://github.com/Maratyszcza/psimd.git) registered for path 'third_party/psimd'
2022-12-01T10:01:37.9689606Z Submodule 'third_party/NNPACK_deps/pthreadpool' (https://github.com/Maratyszcza/pthreadpool.git) registered for path 'third_party/pthreadpool'
2022-12-01T10:01:37.9695216Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/pybind11'
2022-12-01T10:01:37.9700997Z Submodule 'third_party/python-enum' (https://github.com/PeachPy/enum34.git) registered for path 'third_party/python-enum'
2022-12-01T10:01:37.9707135Z Submodule 'third_party/python-peachpy' (https://github.com/malfet/PeachPy.git) registered for path 'third_party/python-peachpy'
2022-12-01T10:01:37.9712671Z Submodule 'third_party/python-six' (https://github.com/benjaminp/six.git) registered for path 'third_party/python-six'
2022-12-01T10:01:37.9718383Z Submodule 'third_party/sleef' (https://github.com/shibatch/sleef) registered for path 'third_party/sleef'
2022-12-01T10:01:37.9724944Z Submodule 'third_party/tbb' (https://github.com/01org/tbb) registered for path 'third_party/tbb'
2022-12-01T10:01:37.9731136Z Submodule 'third_party/tensorpipe' (https://github.com/pytorch/tensorpipe.git) registered for path 'third_party/tensorpipe'
2022-12-01T10:01:37.9737370Z Submodule 'third_party/zstd' (https://github.com/facebook/zstd.git) registered for path 'third_party/zstd'
2022-12-01T10:01:37.9763967Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/android/libs/fbjni'...
2022-12-01T10:01:38.2819096Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FP16'...
2022-12-01T10:01:38.4996762Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FXdiv'...
2022-12-01T10:01:38.7229648Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/NNPACK'...
2022-12-01T10:01:39.0290321Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/QNNPACK'...
2022-12-01T10:01:39.3173781Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/VulkanMemoryAllocator'...
2022-12-01T10:01:41.3675954Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/XNNPACK'...
2022-12-01T10:01:47.3009544Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/benchmark'...
2022-12-01T10:01:47.6794605Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cpuinfo'...
2022-12-01T10:01:48.2214748Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cub'...
2022-12-01T10:01:49.6964222Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cudnn_frontend'...
2022-12-01T10:01:50.7881539Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cutlass'...
2022-12-01T10:01:52.1191395Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/eigen'...
2022-12-01T10:01:57.2123182Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm'...
2022-12-01T10:01:57.9987782Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/flatbuffers'...
2022-12-01T10:01:59.1631445Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fmt'...
2022-12-01T10:02:00.3963575Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/foxi'...
2022-12-01T10:02:00.6214788Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gemmlowp/gemmlowp'...
2022-12-01T10:02:01.2385736Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gloo'...
2022-12-01T10:02:01.9839599Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/googletest'...
2022-12-01T10:02:03.4984939Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep'...
2022-12-01T10:02:03.9866960Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ios-cmake'...
2022-12-01T10:02:04.3250609Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ittapi'...
2022-12-01T10:02:04.6285613Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto'...
2022-12-01T10:02:06.0456282Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nccl/nccl'...
2022-12-01T10:02:06.5789069Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/neon2sse'...
2022-12-01T10:02:07.0015049Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nlohmann'...
2022-12-01T10:02:13.9205384Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx'...
2022-12-01T10:02:15.5312508Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt'...
2022-12-01T10:02:16.9954396Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pocketfft'...
2022-12-01T10:02:17.2222464Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf'...
2022-12-01T10:02:22.6433883Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/psimd'...
2022-12-01T10:02:22.8512084Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pthreadpool'...
2022-12-01T10:02:23.0957799Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pybind11'...
2022-12-01T10:02:23.8892120Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/python-enum'...
2022-12-01T10:02:24.4261703Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/python-peachpy'...
2022-12-01T10:02:24.7936233Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/python-six'...
2022-12-01T10:02:25.0658019Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/sleef'...
2022-12-01T10:02:25.6374943Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tbb'...
2022-12-01T10:02:27.8639202Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe'...
2022-12-01T10:02:28.5898766Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/zstd'...
2022-12-01T10:02:30.7066813Z Submodule path 'android/libs/fbjni': checked out '7e1e1fe3858c63c251c637ae41a20de425dde96f'
2022-12-01T10:02:30.7163042Z Submodule path 'third_party/FP16': checked out '4dfe081cf6bcd15db339cf2680b9281b8451eeb3'
2022-12-01T10:02:30.7238754Z Submodule path 'third_party/FXdiv': checked out 'b408327ac2a15ec3e43352421954f5b1967701d1'
2022-12-01T10:02:30.7447966Z Submodule path 'third_party/NNPACK': checked out 'c07e3a0400713d546e0dea2d5466dd22ea389c73'
2022-12-01T10:02:30.7653231Z Submodule path 'third_party/QNNPACK': checked out '7d2a4e9931a82adc3814275b6219a03e24e36b4c'
2022-12-01T10:02:30.7980096Z Submodule path 'third_party/VulkanMemoryAllocator': checked out 'a6bfc237255a6bac1513f7c1ebde6d8aed6b5191'
2022-12-01T10:02:31.3507264Z Submodule path 'third_party/XNNPACK': checked out 'ae108ef49aa5623b896fc93d4298c49d1750d9ba'
2022-12-01T10:02:31.3701081Z Submodule path 'third_party/benchmark': checked out '0d98dba29d66e93259db7daa53a9327df767a415'
2022-12-01T10:02:31.4612339Z Submodule path 'third_party/cpuinfo': checked out '8ec7bd91ad0470e61cf38f618cc1f270dede599c'
2022-12-01T10:02:31.4932486Z Submodule path 'third_party/cub': checked out 'd106ddb991a56c3df1b6d51b2409e36ba8181ce4'
2022-12-01T10:02:31.7593119Z Submodule path 'third_party/cudnn_frontend': checked out '171a7a986f7fbd9ed71bd0cf3c7ad4f55843d6b3'
2022-12-01T10:02:32.1345995Z Submodule path 'third_party/cutlass': checked out 'b72cbf957df8cf84a6d0ff91c190ad51a9c1d24a'
2022-12-01T10:02:32.3580015Z Submodule path 'third_party/eigen': checked out '3147391d946bb4b6c68edd901f2add6ac1f31f8c'
2022-12-01T10:02:32.4001730Z Submodule path 'third_party/fbgemm': checked out '0d98c261561524cce92e37fe307ea6596664309a'
2022-12-01T10:02:32.4016484Z Submodule 'third_party/asmjit' (https://github.com/asmjit/asmjit.git) registered for path 'third_party/fbgemm/third_party/asmjit'
2022-12-01T10:02:32.4018630Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo) registered for path 'third_party/fbgemm/third_party/cpuinfo'
2022-12-01T10:02:32.4021446Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/fbgemm/third_party/googletest'
2022-12-01T10:02:32.4024162Z Submodule 'third_party/hipify_torch' (https://github.com/ROCmSoftwarePlatform/hipify_torch.git) registered for path 'third_party/fbgemm/third_party/hipify_torch'
2022-12-01T10:02:32.4045754Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/asmjit'...
2022-12-01T10:02:33.3675190Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/cpuinfo'...
2022-12-01T10:02:33.8943432Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/googletest'...
2022-12-01T10:02:35.0652922Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/hipify_torch'...
2022-12-01T10:02:35.3896738Z Submodule path 'third_party/fbgemm/third_party/asmjit': checked out 'd3fbf7c9bc7c1d1365a94a45614b91c5a3706b81'
2022-12-01T10:02:35.4815799Z Submodule path 'third_party/fbgemm/third_party/cpuinfo': checked out 'ed8b86a253800bafdb7b25c5c399f91bff9cb1f3'
2022-12-01T10:02:35.5362060Z Submodule path 'third_party/fbgemm/third_party/googletest': checked out 'cbf019de22c8dd37b2108da35b2748fd702d1796'
2022-12-01T10:02:35.5456454Z Submodule path 'third_party/fbgemm/third_party/hipify_torch': checked out '1840658c184f3eeba787dae0f06c45756c1daaf5'
2022-12-01T10:02:35.6238514Z Submodule path 'third_party/flatbuffers': checked out 'd0cede9c90c5257537c293517a21376408b549fa'
2022-12-01T10:02:35.6569830Z Submodule path 'third_party/fmt': checked out 'cd4af11efc9c622896a3e4cb599fa28668ca3d05'
2022-12-01T10:02:35.6652515Z Submodule path 'third_party/foxi': checked out 'c278588e34e535f0bb8f00df3880d26928038cad'
2022-12-01T10:02:35.7020818Z Submodule path 'third_party/gemmlowp/gemmlowp': checked out '3fb5c176c17c765a3492cd2f0321b0dab712f350'
2022-12-01T10:02:35.7242528Z Submodule path 'third_party/gloo': checked out '5b143513263133af2b95547e97c07cebeb72bf72'
2022-12-01T10:02:35.7678073Z Submodule path 'third_party/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929'
2022-12-01T10:02:35.7785638Z Submodule path 'third_party/ideep': checked out '77d662b313a762e82b389d3fd965e0098f12cd99'
2022-12-01T10:02:35.7797785Z Submodule 'mkl-dnn' (https://github.com/intel/mkl-dnn.git) registered for path 'third_party/ideep/mkl-dnn'
2022-12-01T10:02:35.7819057Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep/mkl-dnn'...
2022-12-01T10:02:43.3612408Z Submodule path 'third_party/ideep/mkl-dnn': checked out '888a87a954e4fddb4d81fd10858eb834f2441b46'
2022-12-01T10:02:43.3628382Z Submodule 'third_party/oneDNN' (https://github.com/oneapi-src/oneDNN.git) registered for path 'third_party/ideep/mkl-dnn/third_party/oneDNN'
2022-12-01T10:02:43.3651040Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep/mkl-dnn/third_party/oneDNN'...
2022-12-01T10:02:50.9344735Z Submodule path 'third_party/ideep/mkl-dnn/third_party/oneDNN': checked out '52b5f107dd9cf10910aaa19cb47f3abf9b349815'
2022-12-01T10:02:50.9440485Z Submodule path 'third_party/ios-cmake': checked out '8abaed637d56f1337d6e1d2c4026e25c1eade724'
2022-12-01T10:02:50.9572915Z Submodule path 'third_party/ittapi': checked out '5b8a7d7422611c3a0d799fb5fc5dd4abfae35b42'
2022-12-01T10:02:51.0440521Z Submodule path 'third_party/kineto': checked out '0703c78999061b8329dfab7ec5046fc5764a5573'
2022-12-01T10:02:51.0455498Z Submodule 'libkineto/third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/fmt'
2022-12-01T10:02:51.0456428Z Submodule 'libkineto/third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/googletest'
2022-12-01T10:02:51.0479989Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/fmt'...
2022-12-01T10:02:52.2251882Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/googletest'...
2022-12-01T10:02:53.3422803Z Submodule path 'third_party/kineto/libkineto/third_party/fmt': checked out '2591ab91c3898c9f6544fff04660276537d32ffd'
2022-12-01T10:02:53.3930834Z Submodule path 'third_party/kineto/libkineto/third_party/googletest': checked out '7aca84427f224eeed3144123d5230d5871e93347'
2022-12-01T10:02:53.4117801Z Submodule path 'third_party/nccl/nccl': checked out 'f89fd4777d2ef9229c039ff750ae21da01626f52'
2022-12-01T10:02:53.4246477Z Submodule path 'third_party/neon2sse': checked out '97a126f08ce318023be604d03f88bf0820a9464a'
2022-12-01T10:02:53.5231415Z Submodule path 'third_party/nlohmann': checked out '87cda1d6646592ac5866dc703c8e1839046a6806'
2022-12-01T10:02:53.7477206Z Submodule path 'third_party/onnx': checked out 'f7ee1ac60d06abe8e26c9b6bbe1e3db5286b614b'
2022-12-01T10:02:53.7503369Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/onnx/third_party/benchmark'
2022-12-01T10:02:53.7505137Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/onnx/third_party/pybind11'
2022-12-01T10:02:53.7528467Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/benchmark'...
2022-12-01T10:02:55.1786965Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/pybind11'...
2022-12-01T10:02:55.9760240Z Submodule path 'third_party/onnx/third_party/benchmark': checked out '0d98dba29d66e93259db7daa53a9327df767a415'
2022-12-01T10:02:56.0048519Z Submodule path 'third_party/onnx/third_party/pybind11': checked out 'ffa346860b306c9bbfb341aed9c14c067751feb8'
2022-12-01T10:02:56.0187406Z Submodule path 'third_party/onnx-tensorrt': checked out 'c153211418a7c57ce071d9ce2a41f8d1c85a878f'
2022-12-01T10:02:56.0200192Z Submodule 'third_party/onnx' (https://github.com/onnx/onnx.git) registered for path 'third_party/onnx-tensorrt/third_party/onnx'
2022-12-01T10:02:56.0221439Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt/third_party/onnx'...
2022-12-01T10:02:57.7598694Z Submodule path 'third_party/onnx-tensorrt/third_party/onnx': checked out '765f5ee823a67a866f4bd28a9860e81f3c811ce8'
2022-12-01T10:02:57.7618116Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'
2022-12-01T10:02:57.7620088Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'
2022-12-01T10:02:57.7643267Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'...
2022-12-01T10:02:58.1590833Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'...
2022-12-01T10:02:59.0102868Z Submodule path 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark': checked out 'e776aa0275e293707b6a0901e0e8d8a8a3679508'
2022-12-01T10:02:59.0707857Z Submodule path 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11': checked out 'a1041190c8b8ff0cd9e2f0752248ad5e3789ea0c'
2022-12-01T10:02:59.0722554Z Submodule 'tools/clang' (https://github.com/wjakob/clang-cindex-python3) registered for path 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'
2022-12-01T10:02:59.0746144Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'...
2022-12-01T10:02:59.2915068Z Submodule path 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang': checked out '6a00cbc4a9b8e68b71caf7f774b3f9c753ae84d5'
2022-12-01T10:02:59.2996199Z Submodule path 'third_party/pocketfft': checked out 'ea778e37710c07723435b1be58235996d1d43a5a'
2022-12-01T10:02:59.5348933Z Submodule path 'third_party/protobuf': checked out 'd1eca4e4b421cd2997495c4b4e65cea6be4e9b8a'
2022-12-01T10:02:59.5366540Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/protobuf/third_party/benchmark'
2022-12-01T10:02:59.5368145Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/protobuf/third_party/googletest'
2022-12-01T10:02:59.5391837Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/benchmark'...
2022-12-01T10:02:59.9483341Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/googletest'...
2022-12-01T10:03:01.0449701Z Submodule path 'third_party/protobuf/third_party/benchmark': checked out '5b7683f49e1e9223cf9927b24f6fd3d6bd82e3f8'
2022-12-01T10:03:01.1081900Z Submodule path 'third_party/protobuf/third_party/googletest': checked out '5ec7f0c4a113e2f18ac2c6cc7df51ad6afc24081'
2022-12-01T10:03:01.1161283Z Submodule path 'third_party/psimd': checked out '072586a71b55b7f8c584153d223e95687148a900'
2022-12-01T10:03:01.1260300Z Submodule path 'third_party/pthreadpool': checked out 'a134dd5d4cee80cce15db81a72e7f929d71dd413'
2022-12-01T10:03:01.1561602Z Submodule path 'third_party/pybind11': checked out 'aa304c9c7d725ffb9d10af08a3b34cb372307020'
2022-12-01T10:03:01.1644167Z Submodule path 'third_party/python-enum': checked out '4cfedc426c4e2fc52e3f5c2b4297e15ed8d6b8c7'
2022-12-01T10:03:01.1904499Z Submodule path 'third_party/python-peachpy': checked out 'f45429b087dd7d5bc78bb40dc7cf06425c252d67'
2022-12-01T10:03:01.1990866Z Submodule path 'third_party/python-six': checked out '15e31431af97e5e64b80af0a3f598d382bcdd49a'
2022-12-01T10:03:01.2393073Z Submodule path 'third_party/sleef': checked out 'e0a003ee838b75d11763aa9c3ef17bf71a725bff'
2022-12-01T10:03:01.3388370Z Submodule path 'third_party/tbb': checked out 'a51a90bc609bb73db8ea13841b5cf7aa4344d4a9'
2022-12-01T10:03:01.3627589Z Submodule path 'third_party/tensorpipe': checked out '52791a2fd214b2a9dc5759d36725909c1daa7f2e'
2022-12-01T10:03:01.3641685Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/tensorpipe/third_party/googletest'
2022-12-01T10:03:01.3643820Z Submodule 'third_party/libnop' (https://github.com/google/libnop.git) registered for path 'third_party/tensorpipe/third_party/libnop'
2022-12-01T10:03:01.3646209Z Submodule 'third_party/libuv' (https://github.com/libuv/libuv.git) registered for path 'third_party/tensorpipe/third_party/libuv'
2022-12-01T10:03:01.3648747Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/tensorpipe/third_party/pybind11'
2022-12-01T10:03:01.3671533Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/googletest'...
2022-12-01T10:03:02.6887428Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libnop'...
2022-12-01T10:03:03.2175759Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libuv'...
2022-12-01T10:03:04.9240610Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11'...
2022-12-01T10:03:05.8945757Z Submodule path 'third_party/tensorpipe/third_party/googletest': checked out 'aee0f9d9b5b87796ee8a0ab26b7587ec30e8858e'
2022-12-01T10:03:05.9078690Z Submodule path 'third_party/tensorpipe/third_party/libnop': checked out '910b55815be16109f04f4180e9adee14fb4ce281'
2022-12-01T10:03:05.9679113Z Submodule path 'third_party/tensorpipe/third_party/libuv': checked out '1dff88e5161cba5c59276d2070d2e304e4dcb242'
2022-12-01T10:03:05.9938735Z Submodule path 'third_party/tensorpipe/third_party/pybind11': checked out 'a23996fce38ff6ccfbcdc09f1e63f2c4be5ea2ef'
2022-12-01T10:03:05.9952930Z Submodule 'tools/clang' (https://github.com/wjakob/clang-cindex-python3) registered for path 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2022-12-01T10:03:05.9974319Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11/tools/clang'...
2022-12-01T10:03:06.2586972Z Submodule path 'third_party/tensorpipe/third_party/pybind11/tools/clang': checked out '6a00cbc4a9b8e68b71caf7f774b3f9c753ae84d5'
2022-12-01T10:03:06.3821102Z Submodule path 'third_party/zstd': checked out 'aec56a52fbab207fc639a1937d1e708a282edca8'
2022-12-01T10:03:06.3849408Z [command]/usr/bin/git submodule foreach --recursive git config --local gc.auto 0
2022-12-01T10:03:06.4111281Z Entering 'android/libs/fbjni'
2022-12-01T10:03:06.4146195Z Entering 'third_party/FP16'
2022-12-01T10:03:06.4179602Z Entering 'third_party/FXdiv'
2022-12-01T10:03:06.4214491Z Entering 'third_party/NNPACK'
2022-12-01T10:03:06.4249379Z Entering 'third_party/QNNPACK'
2022-12-01T10:03:06.4284102Z Entering 'third_party/VulkanMemoryAllocator'
2022-12-01T10:03:06.4319073Z Entering 'third_party/XNNPACK'
2022-12-01T10:03:06.4363635Z Entering 'third_party/benchmark'
2022-12-01T10:03:06.4398456Z Entering 'third_party/cpuinfo'
2022-12-01T10:03:06.4433216Z Entering 'third_party/cub'
2022-12-01T10:03:06.4468380Z Entering 'third_party/cudnn_frontend'
2022-12-01T10:03:06.4507959Z Entering 'third_party/cutlass'
2022-12-01T10:03:06.4550075Z Entering 'third_party/eigen'
2022-12-01T10:03:06.4587819Z Entering 'third_party/fbgemm'
2022-12-01T10:03:06.4623172Z Entering 'third_party/fbgemm/third_party/asmjit'
2022-12-01T10:03:06.4655913Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2022-12-01T10:03:06.4690871Z Entering 'third_party/fbgemm/third_party/googletest'
2022-12-01T10:03:06.4726626Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2022-12-01T10:03:06.4761804Z Entering 'third_party/flatbuffers'
2022-12-01T10:03:06.4798703Z Entering 'third_party/fmt'
2022-12-01T10:03:06.4832686Z Entering 'third_party/foxi'
2022-12-01T10:03:06.4867849Z Entering 'third_party/gemmlowp/gemmlowp'
2022-12-01T10:03:06.4902977Z Entering 'third_party/gloo'
2022-12-01T10:03:06.4936731Z Entering 'third_party/googletest'
2022-12-01T10:03:06.4971423Z Entering 'third_party/ideep'
2022-12-01T10:03:06.5004916Z Entering 'third_party/ideep/mkl-dnn'
2022-12-01T10:03:06.5040286Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN'
2022-12-01T10:03:06.5081968Z Entering 'third_party/ios-cmake'
2022-12-01T10:03:06.5116131Z Entering 'third_party/ittapi'
2022-12-01T10:03:06.5149821Z Entering 'third_party/kineto'
2022-12-01T10:03:06.5183763Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2022-12-01T10:03:06.5217161Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2022-12-01T10:03:06.5253236Z Entering 'third_party/nccl/nccl'
2022-12-01T10:03:06.5287030Z Entering 'third_party/neon2sse'
2022-12-01T10:03:06.5321393Z Entering 'third_party/nlohmann'
2022-12-01T10:03:06.5357212Z Entering 'third_party/onnx'
2022-12-01T10:03:06.5404666Z Entering 'third_party/onnx/third_party/benchmark'
2022-12-01T10:03:06.5438694Z Entering 'third_party/onnx/third_party/pybind11'
2022-12-01T10:03:06.5475166Z Entering 'third_party/onnx-tensorrt'
2022-12-01T10:03:06.5509323Z Entering 'third_party/onnx-tensorrt/third_party/onnx'
2022-12-01T10:03:06.5547751Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'
2022-12-01T10:03:06.5583004Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'
2022-12-01T10:03:06.5616665Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'
2022-12-01T10:03:06.5655102Z Entering 'third_party/pocketfft'
2022-12-01T10:03:06.5690475Z Entering 'third_party/protobuf'
2022-12-01T10:03:06.5728599Z Entering 'third_party/protobuf/third_party/benchmark'
2022-12-01T10:03:06.5762911Z Entering 'third_party/protobuf/third_party/googletest'
2022-12-01T10:03:06.5799985Z Entering 'third_party/psimd'
2022-12-01T10:03:06.5834936Z Entering 'third_party/pthreadpool'
2022-12-01T10:03:06.5868793Z Entering 'third_party/pybind11'
2022-12-01T10:03:06.5904162Z Entering 'third_party/python-enum'
2022-12-01T10:03:06.5938899Z Entering 'third_party/python-peachpy'
2022-12-01T10:03:06.5973061Z Entering 'third_party/python-six'
2022-12-01T10:03:06.6006547Z Entering 'third_party/sleef'
2022-12-01T10:03:06.6040808Z Entering 'third_party/tbb'
2022-12-01T10:03:06.6076957Z Entering 'third_party/tensorpipe'
2022-12-01T10:03:06.6111145Z Entering 'third_party/tensorpipe/third_party/googletest'
2022-12-01T10:03:06.6144681Z Entering 'third_party/tensorpipe/third_party/libnop'
2022-12-01T10:03:06.6177442Z Entering 'third_party/tensorpipe/third_party/libuv'
2022-12-01T10:03:06.6210679Z Entering 'third_party/tensorpipe/third_party/pybind11'
2022-12-01T10:03:06.6243499Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2022-12-01T10:03:06.6279572Z Entering 'third_party/zstd'
2022-12-01T10:03:06.6323568Z ##[endgroup]
2022-12-01T10:03:06.6323998Z ##[group]Persisting credentials for submodules
2022-12-01T10:03:06.6328473Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'url\.https\:\/\/github\.com\/\.insteadOf' && git config --local --unset-all 'url.https://github.com/.insteadOf' || :
2022-12-01T10:03:06.6583728Z Entering 'android/libs/fbjni'
2022-12-01T10:03:06.6617435Z Entering 'third_party/FP16'
2022-12-01T10:03:06.6652479Z Entering 'third_party/FXdiv'
2022-12-01T10:03:06.6685689Z Entering 'third_party/NNPACK'
2022-12-01T10:03:06.6719857Z Entering 'third_party/QNNPACK'
2022-12-01T10:03:06.6754625Z Entering 'third_party/VulkanMemoryAllocator'
2022-12-01T10:03:06.6788784Z Entering 'third_party/XNNPACK'
2022-12-01T10:03:06.6832713Z Entering 'third_party/benchmark'
2022-12-01T10:03:06.6867674Z Entering 'third_party/cpuinfo'
2022-12-01T10:03:06.6902500Z Entering 'third_party/cub'
2022-12-01T10:03:06.6936820Z Entering 'third_party/cudnn_frontend'
2022-12-01T10:03:06.6977268Z Entering 'third_party/cutlass'
2022-12-01T10:03:06.7017772Z Entering 'third_party/eigen'
2022-12-01T10:03:06.7055036Z Entering 'third_party/fbgemm'
2022-12-01T10:03:06.7088724Z Entering 'third_party/fbgemm/third_party/asmjit'
2022-12-01T10:03:06.7121396Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2022-12-01T10:03:06.7154448Z Entering 'third_party/fbgemm/third_party/googletest'
2022-12-01T10:03:06.7187808Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2022-12-01T10:03:06.7222183Z Entering 'third_party/flatbuffers'
2022-12-01T10:03:06.7258001Z Entering 'third_party/fmt'
2022-12-01T10:03:06.7292395Z Entering 'third_party/foxi'
2022-12-01T10:03:06.7327023Z Entering 'third_party/gemmlowp/gemmlowp'
2022-12-01T10:03:06.7361271Z Entering 'third_party/gloo'
2022-12-01T10:03:06.7396419Z Entering 'third_party/googletest'
2022-12-01T10:03:06.7430225Z Entering 'third_party/ideep'
2022-12-01T10:03:06.7463182Z Entering 'third_party/ideep/mkl-dnn'
2022-12-01T10:03:06.7497988Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN'
2022-12-01T10:03:06.7537375Z Entering 'third_party/ios-cmake'
2022-12-01T10:03:06.7571241Z Entering 'third_party/ittapi'
2022-12-01T10:03:06.7604042Z Entering 'third_party/kineto'
2022-12-01T10:03:06.7637462Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2022-12-01T10:03:06.7671124Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2022-12-01T10:03:06.7706053Z Entering 'third_party/nccl/nccl'
2022-12-01T10:03:06.7740267Z Entering 'third_party/neon2sse'
2022-12-01T10:03:06.7772984Z Entering 'third_party/nlohmann'
2022-12-01T10:03:06.7808023Z Entering 'third_party/onnx'
2022-12-01T10:03:06.7856616Z Entering 'third_party/onnx/third_party/benchmark'
2022-12-01T10:03:06.7891333Z Entering 'third_party/onnx/third_party/pybind11'
2022-12-01T10:03:06.7926720Z Entering 'third_party/onnx-tensorrt'
2022-12-01T10:03:06.7961524Z Entering 'third_party/onnx-tensorrt/third_party/onnx'
2022-12-01T10:03:06.7998493Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'
2022-12-01T10:03:06.8032227Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'
2022-12-01T10:03:06.8065509Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'
2022-12-01T10:03:06.8104813Z Entering 'third_party/pocketfft'
2022-12-01T10:03:06.8138490Z Entering 'third_party/protobuf'
2022-12-01T10:03:06.8175791Z Entering 'third_party/protobuf/third_party/benchmark'
2022-12-01T10:03:06.8209813Z Entering 'third_party/protobuf/third_party/googletest'
2022-12-01T10:03:06.8244916Z Entering 'third_party/psimd'
2022-12-01T10:03:06.8278545Z Entering 'third_party/pthreadpool'
2022-12-01T10:03:06.8311216Z Entering 'third_party/pybind11'
2022-12-01T10:03:06.8347287Z Entering 'third_party/python-enum'
2022-12-01T10:03:06.8380081Z Entering 'third_party/python-peachpy'
2022-12-01T10:03:06.8412836Z Entering 'third_party/python-six'
2022-12-01T10:03:06.8446502Z Entering 'third_party/sleef'
2022-12-01T10:03:06.8480712Z Entering 'third_party/tbb'
2022-12-01T10:03:06.8516283Z Entering 'third_party/tensorpipe'
2022-12-01T10:03:06.8550457Z Entering 'third_party/tensorpipe/third_party/googletest'
2022-12-01T10:03:06.8583624Z Entering 'third_party/tensorpipe/third_party/libnop'
2022-12-01T10:03:06.8616066Z Entering 'third_party/tensorpipe/third_party/libuv'
2022-12-01T10:03:06.8649394Z Entering 'third_party/tensorpipe/third_party/pybind11'
2022-12-01T10:03:06.8682381Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2022-12-01T10:03:06.8720686Z Entering 'third_party/zstd'
2022-12-01T10:03:06.8765990Z [command]/usr/bin/git submodule foreach --recursive git config --local 'http.https://github.com/.extraheader' 'AUTHORIZATION: basic ***' && git config --local --show-origin --name-only --get-regexp remote.origin.url
2022-12-01T10:03:06.9025426Z Entering 'android/libs/fbjni'
2022-12-01T10:03:06.9056975Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/android/libs/fbjni/config	remote.origin.url
2022-12-01T10:03:06.9070680Z Entering 'third_party/FP16'
2022-12-01T10:03:06.9103480Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FP16/config	remote.origin.url
2022-12-01T10:03:06.9116838Z Entering 'third_party/FXdiv'
2022-12-01T10:03:06.9149035Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FXdiv/config	remote.origin.url
2022-12-01T10:03:06.9162352Z Entering 'third_party/NNPACK'
2022-12-01T10:03:06.9193499Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK/config	remote.origin.url
2022-12-01T10:03:06.9207105Z Entering 'third_party/QNNPACK'
2022-12-01T10:03:06.9239482Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/QNNPACK/config	remote.origin.url
2022-12-01T10:03:06.9253448Z Entering 'third_party/VulkanMemoryAllocator'
2022-12-01T10:03:06.9285313Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/VulkanMemoryAllocator/config	remote.origin.url
2022-12-01T10:03:06.9298760Z Entering 'third_party/XNNPACK'
2022-12-01T10:03:06.9331116Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/XNNPACK/config	remote.origin.url
2022-12-01T10:03:06.9355476Z Entering 'third_party/benchmark'
2022-12-01T10:03:06.9387815Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/benchmark/config	remote.origin.url
2022-12-01T10:03:06.9401919Z Entering 'third_party/cpuinfo'
2022-12-01T10:03:06.9433681Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpuinfo/config	remote.origin.url
2022-12-01T10:03:06.9448773Z Entering 'third_party/cub'
2022-12-01T10:03:06.9480577Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cub/config	remote.origin.url
2022-12-01T10:03:06.9494760Z Entering 'third_party/cudnn_frontend'
2022-12-01T10:03:06.9526110Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cudnn_frontend/config	remote.origin.url
2022-12-01T10:03:06.9545594Z Entering 'third_party/cutlass'
2022-12-01T10:03:06.9576743Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cutlass/config	remote.origin.url
2022-12-01T10:03:06.9599606Z Entering 'third_party/eigen'
2022-12-01T10:03:06.9630831Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/eigen/config	remote.origin.url
2022-12-01T10:03:06.9646372Z Entering 'third_party/fbgemm'
2022-12-01T10:03:06.9679091Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/config	remote.origin.url
2022-12-01T10:03:06.9693196Z Entering 'third_party/fbgemm/third_party/asmjit'
2022-12-01T10:03:06.9724269Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/asmjit/config	remote.origin.url
2022-12-01T10:03:06.9738749Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2022-12-01T10:03:06.9771032Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cpuinfo/config	remote.origin.url
2022-12-01T10:03:06.9785365Z Entering 'third_party/fbgemm/third_party/googletest'
2022-12-01T10:03:06.9816258Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/googletest/config	remote.origin.url
2022-12-01T10:03:06.9829928Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2022-12-01T10:03:06.9862151Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/hipify_torch/config	remote.origin.url
2022-12-01T10:03:06.9876349Z Entering 'third_party/flatbuffers'
2022-12-01T10:03:06.9910476Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flatbuffers/config	remote.origin.url
2022-12-01T10:03:06.9926562Z Entering 'third_party/fmt'
2022-12-01T10:03:06.9957990Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fmt/config	remote.origin.url
2022-12-01T10:03:06.9971643Z Entering 'third_party/foxi'
2022-12-01T10:03:07.0004372Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/foxi/config	remote.origin.url
2022-12-01T10:03:07.0018064Z Entering 'third_party/gemmlowp/gemmlowp'
2022-12-01T10:03:07.0052146Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gemmlowp/gemmlowp/config	remote.origin.url
2022-12-01T10:03:07.0066575Z Entering 'third_party/gloo'
2022-12-01T10:03:07.0098531Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gloo/config	remote.origin.url
2022-12-01T10:03:07.0113196Z Entering 'third_party/googletest'
2022-12-01T10:03:07.0145613Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/googletest/config	remote.origin.url
2022-12-01T10:03:07.0160201Z Entering 'third_party/ideep'
2022-12-01T10:03:07.0191009Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/config	remote.origin.url
2022-12-01T10:03:07.0204653Z Entering 'third_party/ideep/mkl-dnn'
2022-12-01T10:03:07.0235507Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/modules/mkl-dnn/config	remote.origin.url
2022-12-01T10:03:07.0250845Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN'
2022-12-01T10:03:07.0282121Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/modules/mkl-dnn/modules/third_party/oneDNN/config	remote.origin.url
2022-12-01T10:03:07.0303545Z Entering 'third_party/ios-cmake'
2022-12-01T10:03:07.0336277Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ios-cmake/config	remote.origin.url
2022-12-01T10:03:07.0350181Z Entering 'third_party/ittapi'
2022-12-01T10:03:07.0381497Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ittapi/config	remote.origin.url
2022-12-01T10:03:07.0395253Z Entering 'third_party/kineto'
2022-12-01T10:03:07.0427272Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/config	remote.origin.url
2022-12-01T10:03:07.0441755Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2022-12-01T10:03:07.0473275Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/fmt/config	remote.origin.url
2022-12-01T10:03:07.0486859Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2022-12-01T10:03:07.0518522Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/googletest/config	remote.origin.url
2022-12-01T10:03:07.0533641Z Entering 'third_party/nccl/nccl'
2022-12-01T10:03:07.0565629Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nccl/nccl/config	remote.origin.url
2022-12-01T10:03:07.0579583Z Entering 'third_party/neon2sse'
2022-12-01T10:03:07.0611601Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/neon2sse/config	remote.origin.url
2022-12-01T10:03:07.0626710Z Entering 'third_party/nlohmann'
2022-12-01T10:03:07.0659076Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nlohmann/config	remote.origin.url
2022-12-01T10:03:07.0674697Z Entering 'third_party/onnx'
2022-12-01T10:03:07.0708244Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/config	remote.origin.url
2022-12-01T10:03:07.0734147Z Entering 'third_party/onnx/third_party/benchmark'
2022-12-01T10:03:07.0765925Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/benchmark/config	remote.origin.url
2022-12-01T10:03:07.0782343Z Entering 'third_party/onnx/third_party/pybind11'
2022-12-01T10:03:07.0812993Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/pybind11/config	remote.origin.url
2022-12-01T10:03:07.0829805Z Entering 'third_party/onnx-tensorrt'
2022-12-01T10:03:07.0862420Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/config	remote.origin.url
2022-12-01T10:03:07.0876155Z Entering 'third_party/onnx-tensorrt/third_party/onnx'
2022-12-01T10:03:07.0907467Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/modules/third_party/onnx/config	remote.origin.url
2022-12-01T10:03:07.0925324Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'
2022-12-01T10:03:07.0958126Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/modules/third_party/onnx/modules/third_party/benchmark/config	remote.origin.url
2022-12-01T10:03:07.0971523Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'
2022-12-01T10:03:07.1004019Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/modules/third_party/onnx/modules/third_party/pybind11/config	remote.origin.url
2022-12-01T10:03:07.1018261Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'
2022-12-01T10:03:07.1050478Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/modules/third_party/onnx/modules/third_party/pybind11/modules/tools/clang/config	remote.origin.url
2022-12-01T10:03:07.1069644Z Entering 'third_party/pocketfft'
2022-12-01T10:03:07.1103554Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pocketfft/config	remote.origin.url
2022-12-01T10:03:07.1117375Z Entering 'third_party/protobuf'
2022-12-01T10:03:07.1151341Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/config	remote.origin.url
2022-12-01T10:03:07.1168786Z Entering 'third_party/protobuf/third_party/benchmark'
2022-12-01T10:03:07.1201140Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/benchmark/config	remote.origin.url
2022-12-01T10:03:07.1215507Z Entering 'third_party/protobuf/third_party/googletest'
2022-12-01T10:03:07.1248407Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/googletest/config	remote.origin.url
2022-12-01T10:03:07.1265114Z Entering 'third_party/psimd'
2022-12-01T10:03:07.1296695Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/psimd/config	remote.origin.url
2022-12-01T10:03:07.1310410Z Entering 'third_party/pthreadpool'
2022-12-01T10:03:07.1343769Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/pthreadpool/config	remote.origin.url
2022-12-01T10:03:07.1357598Z Entering 'third_party/pybind11'
2022-12-01T10:03:07.1389001Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pybind11/config	remote.origin.url
2022-12-01T10:03:07.1403517Z Entering 'third_party/python-enum'
2022-12-01T10:03:07.1436786Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-enum/config	remote.origin.url
2022-12-01T10:03:07.1451203Z Entering 'third_party/python-peachpy'
2022-12-01T10:03:07.1484609Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-peachpy/config	remote.origin.url
2022-12-01T10:03:07.1498712Z Entering 'third_party/python-six'
2022-12-01T10:03:07.1531931Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-six/config	remote.origin.url
2022-12-01T10:03:07.1546414Z Entering 'third_party/sleef'
2022-12-01T10:03:07.1578729Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/sleef/config	remote.origin.url
2022-12-01T10:03:07.1594110Z Entering 'third_party/tbb'
2022-12-01T10:03:07.1625703Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tbb/config	remote.origin.url
2022-12-01T10:03:07.1641773Z Entering 'third_party/tensorpipe'
2022-12-01T10:03:07.1673585Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/config	remote.origin.url
2022-12-01T10:03:07.1687850Z Entering 'third_party/tensorpipe/third_party/googletest'
2022-12-01T10:03:07.1719509Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/googletest/config	remote.origin.url
2022-12-01T10:03:07.1733652Z Entering 'third_party/tensorpipe/third_party/libnop'
2022-12-01T10:03:07.1765211Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libnop/config	remote.origin.url
2022-12-01T10:03:07.1778960Z Entering 'third_party/tensorpipe/third_party/libuv'
2022-12-01T10:03:07.1810740Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libuv/config	remote.origin.url
2022-12-01T10:03:07.1825157Z Entering 'third_party/tensorpipe/third_party/pybind11'
2022-12-01T10:03:07.1857317Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/config	remote.origin.url
2022-12-01T10:03:07.1871960Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2022-12-01T10:03:07.1904137Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/modules/tools/clang/config	remote.origin.url
2022-12-01T10:03:07.1920502Z Entering 'third_party/zstd'
2022-12-01T10:03:07.1952895Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/zstd/config	remote.origin.url
2022-12-01T10:03:07.2544442Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'git@github.com:'
2022-12-01T10:03:07.2808125Z Entering 'android/libs/fbjni'
2022-12-01T10:03:07.2843103Z Entering 'third_party/FP16'
2022-12-01T10:03:07.2878923Z Entering 'third_party/FXdiv'
2022-12-01T10:03:07.2913660Z Entering 'third_party/NNPACK'
2022-12-01T10:03:07.2948537Z Entering 'third_party/QNNPACK'
2022-12-01T10:03:07.2984056Z Entering 'third_party/VulkanMemoryAllocator'
2022-12-01T10:03:07.3019126Z Entering 'third_party/XNNPACK'
2022-12-01T10:03:07.3064299Z Entering 'third_party/benchmark'
2022-12-01T10:03:07.3099115Z Entering 'third_party/cpuinfo'
2022-12-01T10:03:07.3134169Z Entering 'third_party/cub'
2022-12-01T10:03:07.3168446Z Entering 'third_party/cudnn_frontend'
2022-12-01T10:03:07.3209092Z Entering 'third_party/cutlass'
2022-12-01T10:03:07.3251827Z Entering 'third_party/eigen'
2022-12-01T10:03:07.3289508Z Entering 'third_party/fbgemm'
2022-12-01T10:03:07.3325739Z Entering 'third_party/fbgemm/third_party/asmjit'
2022-12-01T10:03:07.3360298Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2022-12-01T10:03:07.3394665Z Entering 'third_party/fbgemm/third_party/googletest'
2022-12-01T10:03:07.3430210Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2022-12-01T10:03:07.3465564Z Entering 'third_party/flatbuffers'
2022-12-01T10:03:07.3502510Z Entering 'third_party/fmt'
2022-12-01T10:03:07.3536491Z Entering 'third_party/foxi'
2022-12-01T10:03:07.3571439Z Entering 'third_party/gemmlowp/gemmlowp'
2022-12-01T10:03:07.3607125Z Entering 'third_party/gloo'
2022-12-01T10:03:07.3642463Z Entering 'third_party/googletest'
2022-12-01T10:03:07.3677327Z Entering 'third_party/ideep'
2022-12-01T10:03:07.3711610Z Entering 'third_party/ideep/mkl-dnn'
2022-12-01T10:03:07.3747437Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN'
2022-12-01T10:03:07.3787912Z Entering 'third_party/ios-cmake'
2022-12-01T10:03:07.3822793Z Entering 'third_party/ittapi'
2022-12-01T10:03:07.3857331Z Entering 'third_party/kineto'
2022-12-01T10:03:07.3891531Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2022-12-01T10:03:07.3925379Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2022-12-01T10:03:07.3960898Z Entering 'third_party/nccl/nccl'
2022-12-01T10:03:07.3997084Z Entering 'third_party/neon2sse'
2022-12-01T10:03:07.4031540Z Entering 'third_party/nlohmann'
2022-12-01T10:03:07.4068382Z Entering 'third_party/onnx'
2022-12-01T10:03:07.4114677Z Entering 'third_party/onnx/third_party/benchmark'
2022-12-01T10:03:07.4150083Z Entering 'third_party/onnx/third_party/pybind11'
2022-12-01T10:03:07.4186573Z Entering 'third_party/onnx-tensorrt'
2022-12-01T10:03:07.4222101Z Entering 'third_party/onnx-tensorrt/third_party/onnx'
2022-12-01T10:03:07.4261351Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'
2022-12-01T10:03:07.4295573Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'
2022-12-01T10:03:07.4329945Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'
2022-12-01T10:03:07.4368841Z Entering 'third_party/pocketfft'
2022-12-01T10:03:07.4405842Z Entering 'third_party/protobuf'
2022-12-01T10:03:07.4444915Z Entering 'third_party/protobuf/third_party/benchmark'
2022-12-01T10:03:07.4479746Z Entering 'third_party/protobuf/third_party/googletest'
2022-12-01T10:03:07.4514677Z Entering 'third_party/psimd'
2022-12-01T10:03:07.4549887Z Entering 'third_party/pthreadpool'
2022-12-01T10:03:07.4585791Z Entering 'third_party/pybind11'
2022-12-01T10:03:07.4620094Z Entering 'third_party/python-enum'
2022-12-01T10:03:07.4654681Z Entering 'third_party/python-peachpy'
2022-12-01T10:03:07.4688551Z Entering 'third_party/python-six'
2022-12-01T10:03:07.4723429Z Entering 'third_party/sleef'
2022-12-01T10:03:07.4759544Z Entering 'third_party/tbb'
2022-12-01T10:03:07.4796257Z Entering 'third_party/tensorpipe'
2022-12-01T10:03:07.4832016Z Entering 'third_party/tensorpipe/third_party/googletest'
2022-12-01T10:03:07.4866226Z Entering 'third_party/tensorpipe/third_party/libnop'
2022-12-01T10:03:07.4899932Z Entering 'third_party/tensorpipe/third_party/libuv'
2022-12-01T10:03:07.4966850Z Entering 'third_party/tensorpipe/third_party/pybind11'
2022-12-01T10:03:07.4968263Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2022-12-01T10:03:07.5005413Z Entering 'third_party/zstd'
2022-12-01T10:03:07.5052094Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'org-21003710@github.com:'
2022-12-01T10:03:07.5308957Z Entering 'android/libs/fbjni'
2022-12-01T10:03:07.5344701Z Entering 'third_party/FP16'
2022-12-01T10:03:07.5379541Z Entering 'third_party/FXdiv'
2022-12-01T10:03:07.5414339Z Entering 'third_party/NNPACK'
2022-12-01T10:03:07.5449850Z Entering 'third_party/QNNPACK'
2022-12-01T10:03:07.5484453Z Entering 'third_party/VulkanMemoryAllocator'
2022-12-01T10:03:07.5519055Z Entering 'third_party/XNNPACK'
2022-12-01T10:03:07.5563510Z Entering 'third_party/benchmark'
2022-12-01T10:03:07.5599728Z Entering 'third_party/cpuinfo'
2022-12-01T10:03:07.5635409Z Entering 'third_party/cub'
2022-12-01T10:03:07.5670438Z Entering 'third_party/cudnn_frontend'
2022-12-01T10:03:07.5711058Z Entering 'third_party/cutlass'
2022-12-01T10:03:07.5751483Z Entering 'third_party/eigen'
2022-12-01T10:03:07.5788913Z Entering 'third_party/fbgemm'
2022-12-01T10:03:07.5824715Z Entering 'third_party/fbgemm/third_party/asmjit'
2022-12-01T10:03:07.5859024Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2022-12-01T10:03:07.5900921Z Entering 'third_party/fbgemm/third_party/googletest'
2022-12-01T10:03:07.5929699Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2022-12-01T10:03:07.5965824Z Entering 'third_party/flatbuffers'
2022-12-01T10:03:07.6002313Z Entering 'third_party/fmt'
2022-12-01T10:03:07.6036663Z Entering 'third_party/foxi'
2022-12-01T10:03:07.6070971Z Entering 'third_party/gemmlowp/gemmlowp'
2022-12-01T10:03:07.6106298Z Entering 'third_party/gloo'
2022-12-01T10:03:07.6140705Z Entering 'third_party/googletest'
2022-12-01T10:03:07.6175713Z Entering 'third_party/ideep'
2022-12-01T10:03:07.6210155Z Entering 'third_party/ideep/mkl-dnn'
2022-12-01T10:03:07.6246837Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN'
2022-12-01T10:03:07.6286723Z Entering 'third_party/ios-cmake'
2022-12-01T10:03:07.6321607Z Entering 'third_party/ittapi'
2022-12-01T10:03:07.6356043Z Entering 'third_party/kineto'
2022-12-01T10:03:07.6391287Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2022-12-01T10:03:07.6424833Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2022-12-01T10:03:07.6459515Z Entering 'third_party/nccl/nccl'
2022-12-01T10:03:07.6495855Z Entering 'third_party/neon2sse'
2022-12-01T10:03:07.6529332Z Entering 'third_party/nlohmann'
2022-12-01T10:03:07.6565840Z Entering 'third_party/onnx'
2022-12-01T10:03:07.6612314Z Entering 'third_party/onnx/third_party/benchmark'
2022-12-01T10:03:07.6647520Z Entering 'third_party/onnx/third_party/pybind11'
2022-12-01T10:03:07.6683531Z Entering 'third_party/onnx-tensorrt'
2022-12-01T10:03:07.6718050Z Entering 'third_party/onnx-tensorrt/third_party/onnx'
2022-12-01T10:03:07.6758473Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'
2022-12-01T10:03:07.6791998Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'
2022-12-01T10:03:07.6825893Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'
2022-12-01T10:03:07.6864822Z Entering 'third_party/pocketfft'
2022-12-01T10:03:07.6899343Z Entering 'third_party/protobuf'
2022-12-01T10:03:07.6938379Z Entering 'third_party/protobuf/third_party/benchmark'
2022-12-01T10:03:07.6973000Z Entering 'third_party/protobuf/third_party/googletest'
2022-12-01T10:03:07.7010357Z Entering 'third_party/psimd'
2022-12-01T10:03:07.7047417Z Entering 'third_party/pthreadpool'
2022-12-01T10:03:07.7084018Z Entering 'third_party/pybind11'
2022-12-01T10:03:07.7121255Z Entering 'third_party/python-enum'
2022-12-01T10:03:07.7155253Z Entering 'third_party/python-peachpy'
2022-12-01T10:03:07.7189252Z Entering 'third_party/python-six'
2022-12-01T10:03:07.7223220Z Entering 'third_party/sleef'
2022-12-01T10:03:07.7257604Z Entering 'third_party/tbb'
2022-12-01T10:03:07.7293074Z Entering 'third_party/tensorpipe'
2022-12-01T10:03:07.7327652Z Entering 'third_party/tensorpipe/third_party/googletest'
2022-12-01T10:03:07.7362819Z Entering 'third_party/tensorpipe/third_party/libnop'
2022-12-01T10:03:07.7396090Z Entering 'third_party/tensorpipe/third_party/libuv'
2022-12-01T10:03:07.7431283Z Entering 'third_party/tensorpipe/third_party/pybind11'
2022-12-01T10:03:07.7465238Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2022-12-01T10:03:07.7502689Z Entering 'third_party/zstd'
2022-12-01T10:03:07.7546546Z ##[endgroup]
2022-12-01T10:03:07.7584865Z [command]/usr/bin/git log -1 --format='%H'
2022-12-01T10:03:07.7611767Z 'c13d400bffe90e16b96520bbc8a41a6f0c9cd584'
2022-12-01T10:03:07.7745682Z Prepare all required actions
2022-12-01T10:03:07.7771502Z ##[group]Run ./.github/actions/setup-linux
2022-12-01T10:03:07.7771704Z env:
2022-12-01T10:03:07.7771881Z   GIT_DEFAULT_BRANCH: master
2022-12-01T10:03:07.7772059Z ##[endgroup]
2022-12-01T10:03:07.7787829Z ##[group]Run set -euo pipefail
2022-12-01T10:03:07.7788070Z set -euo pipefail
2022-12-01T10:03:07.7788283Z function get_ec2_metadata() {
2022-12-01T10:03:07.7788517Z   # Pulled from instance metadata endpoint for EC2
2022-12-01T10:03:07.7788870Z   # see https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html
2022-12-01T10:03:07.7789172Z   category=$1
2022-12-01T10:03:07.7789408Z   curl -fsSL "http://169.254.169.254/latest/meta-data/${category}"
2022-12-01T10:03:07.7789630Z }
2022-12-01T10:03:07.7789934Z echo "ami-id: $(get_ec2_metadata ami-id)"
2022-12-01T10:03:07.7790210Z echo "instance-id: $(get_ec2_metadata instance-id)"
2022-12-01T10:03:07.7790493Z echo "instance-type: $(get_ec2_metadata instance-type)"
2022-12-01T10:03:07.7790728Z echo "system info $(uname -a)"
2022-12-01T10:03:07.7802550Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-12-01T10:03:07.7802770Z env:
2022-12-01T10:03:07.7802934Z   GIT_DEFAULT_BRANCH: master
2022-12-01T10:03:07.7803124Z ##[endgroup]
2022-12-01T10:03:07.7883179Z ami-id: ami-096198a0bccc6bad4
2022-12-01T10:03:07.7934175Z instance-id: i-0ede55ee7772c82ca
2022-12-01T10:03:07.7985371Z instance-type: c5.2xlarge
2022-12-01T10:03:07.7992273Z system info Linux ip-10-0-5-106.ec2.internal 4.14.252-195.483.amzn2.x86_64 #1 SMP Mon Nov 1 20:58:46 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
2022-12-01T10:03:07.8011188Z ##[group]Run if systemctl is-active --quiet docker; then
2022-12-01T10:03:07.8011525Z if systemctl is-active --quiet docker; then
2022-12-01T10:03:07.8011807Z     echo "Docker daemon is running...";
2022-12-01T10:03:07.8012054Z else
2022-12-01T10:03:07.8012327Z     echo "Starting docker deamon..." && sudo systemctl start docker;
2022-12-01T10:03:07.8012578Z fi
2022-12-01T10:03:07.8024456Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-12-01T10:03:07.8024711Z env:
2022-12-01T10:03:07.8024908Z   GIT_DEFAULT_BRANCH: master
2022-12-01T10:03:07.8025134Z ##[endgroup]
2022-12-01T10:03:07.8067219Z Docker daemon is running...
2022-12-01T10:03:07.8084025Z ##[group]Run AWS_ACCOUNT_ID=$(aws sts get-caller-identity|grep Account|cut -f4 -d\")
2022-12-01T10:03:07.8084386Z AWS_ACCOUNT_ID=$(aws sts get-caller-identity|grep Account|cut -f4 -d\")
2022-12-01T10:03:07.8084665Z retry () { "$@"  || (sleep 1 && "$@") || (sleep 2 && "$@") }
2022-12-01T10:03:07.8085062Z retry aws ecr get-login*** "$AWS_DEFAULT_REGION" | docker login --username AWS \
2022-12-01T10:03:07.8085417Z     --password-stdin "$AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com"
2022-12-01T10:03:07.8096175Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-12-01T10:03:07.8096377Z env:
2022-12-01T10:03:07.8096549Z   GIT_DEFAULT_BRANCH: master
2022-12-01T10:03:07.8096745Z   AWS_RETRY_MODE: standard
2022-12-01T10:03:07.8096921Z   AWS_MAX_ATTEMPTS: 5
2022-12-01T10:03:07.8097118Z   AWS_DEFAULT_REGION: us-east-1
2022-12-01T10:03:07.8097308Z ##[endgroup]
2022-12-01T10:03:08.6262306Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json.
2022-12-01T10:03:08.6262755Z Configure a credential helper to remove this warning. See
2022-12-01T10:03:08.6263437Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store
2022-12-01T10:03:08.6263813Z 
2022-12-01T10:03:08.6263947Z Login Succeeded
2022-12-01T10:03:08.6292372Z ##[group]Run env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}"
2022-12-01T10:03:08.6292689Z env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}"
2022-12-01T10:03:08.6293040Z env | grep '^CI' >> "/tmp/github_env_${GITHUB_RUN_ID}"
2022-12-01T10:03:08.6304598Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-12-01T10:03:08.6304818Z env:
2022-12-01T10:03:08.6304999Z   GIT_DEFAULT_BRANCH: master
2022-12-01T10:03:08.6305176Z ##[endgroup]
2022-12-01T10:03:08.6378147Z ##[group]Run pytorch/test-infra/.github/actions/setup-ssh@main
2022-12-01T10:03:08.6378383Z with:
2022-12-01T10:03:08.6378973Z   github-secret: ***
2022-12-01T10:03:08.6379180Z   activate-with-label: false
2022-12-01T10:03:08.6379363Z   label: with-ssh
2022-12-01T10:03:08.6379555Z   remove-existing-keys: true
2022-12-01T10:03:08.6379736Z env:
2022-12-01T10:03:08.6379972Z   GIT_DEFAULT_BRANCH: master
2022-12-01T10:03:08.6380157Z ##[endgroup]
2022-12-01T10:03:09.0484068Z Grabbing public ssh keys from https://github.com/charlie-wt.keys
2022-12-01T10:03:09.1390928Z ~/.ssh/authorized_keys file found on node, removing ~/.ssh and starting fresh
2022-12-01T10:03:09.1403840Z Public keys pulled and installed to /home/ec2-user/.ssh/authorized_keys
2022-12-01T10:03:09.1429052Z Login using: ssh ec2-user@ec2-3-214-184-69.compute-1.amazonaws.com
2022-12-01T10:03:09.1491712Z ##[group]Run pytorch/test-infra/.github/actions/pull-docker-image@main
2022-12-01T10:03:09.1491962Z with:
2022-12-01T10:03:09.1492279Z   docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-py3.7-gcc7:fa72f5a0a230eb632055220542038bd4ceca184b
2022-12-01T10:03:09.1492595Z env:
2022-12-01T10:03:09.1492766Z   GIT_DEFAULT_BRANCH: master
2022-12-01T10:03:09.1492943Z ##[endgroup]
2022-12-01T10:03:09.1506740Z ##[group]Run retry () { "$@"  || (sleep 1 && "$@") || (sleep 2 && "$@") }
2022-12-01T10:03:09.1507133Z retry () { "$@"  || (sleep 1 && "$@") || (sleep 2 && "$@") }
2022-12-01T10:03:09.1507406Z # ignore output since only exit code is used for conditional
2022-12-01T10:03:09.1507697Z # only pull docker image if it's not available locally
2022-12-01T10:03:09.1507992Z if ! docker inspect --type=image "${DOCKER_IMAGE}" >/dev/null 2>/dev/null; then
2022-12-01T10:03:09.1508310Z   retry docker pull "${DOCKER_IMAGE}"
2022-12-01T10:03:09.1508511Z fi
2022-12-01T10:03:09.1519941Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-12-01T10:03:09.1520161Z env:
2022-12-01T10:03:09.1520338Z   GIT_DEFAULT_BRANCH: master
2022-12-01T10:03:09.1520680Z   DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-py3.7-gcc7:fa72f5a0a230eb632055220542038bd4ceca184b
2022-12-01T10:03:09.1521013Z ##[endgroup]
2022-12-01T10:03:09.3696591Z fa72f5a0a230eb632055220542038bd4ceca184b: Pulling from pytorch/pytorch-linux-focal-py3.7-gcc7
2022-12-01T10:03:09.3697499Z 675920708c8b: Pulling fs layer
2022-12-01T10:03:09.3698081Z 3a31ffa0162f: Pulling fs layer
2022-12-01T10:03:09.3701042Z fe9b24415086: Pulling fs layer
2022-12-01T10:03:09.3701477Z 422bea31bd2a: Pulling fs layer
2022-12-01T10:03:09.3701825Z 27ad548906a1: Pulling fs layer
2022-12-01T10:03:09.3702110Z efe762c046e3: Pulling fs layer
2022-12-01T10:03:09.3702372Z a12af00c9224: Pulling fs layer
2022-12-01T10:03:09.3702682Z 49809737e89d: Pulling fs layer
2022-12-01T10:03:09.3702877Z 4bd15b741963: Pulling fs layer
2022-12-01T10:03:09.3703470Z 9fdf20fee962: Pulling fs layer
2022-12-01T10:03:09.3703784Z 85d2fa9daf54: Pulling fs layer
2022-12-01T10:03:09.3704013Z e3608d8edfea: Pulling fs layer
2022-12-01T10:03:09.3704212Z 63ae5eb88541: Pulling fs layer
2022-12-01T10:03:09.3704450Z 2b6269e5f012: Pulling fs layer
2022-12-01T10:03:09.3704728Z efe762c046e3: Waiting
2022-12-01T10:03:09.3705023Z 49405b81ba46: Pulling fs layer
2022-12-01T10:03:09.3705346Z 4bd15b741963: Waiting
2022-12-01T10:03:09.3705558Z f5bd51a8525a: Pulling fs layer
2022-12-01T10:03:09.3705851Z 9fdf20fee962: Waiting
2022-12-01T10:03:09.3706144Z dce0658383e3: Pulling fs layer
2022-12-01T10:03:09.3706434Z a12af00c9224: Waiting
2022-12-01T10:03:09.3706690Z fda59f894d4a: Pulling fs layer
2022-12-01T10:03:09.3707022Z c4cd62aabb9c: Pulling fs layer
2022-12-01T10:03:09.3707372Z 85d2fa9daf54: Waiting
2022-12-01T10:03:09.3707703Z d84da0b3b529: Pulling fs layer
2022-12-01T10:03:09.3707967Z 81e2ac298c61: Pulling fs layer
2022-12-01T10:03:09.3708155Z e3608d8edfea: Waiting
2022-12-01T10:03:09.3708328Z 38c05cfe4070: Pulling fs layer
2022-12-01T10:03:09.3708529Z af4d19351fdf: Pulling fs layer
2022-12-01T10:03:09.3708725Z d1d5291b2ab5: Pulling fs layer
2022-12-01T10:03:09.3709032Z 17be6d40a7e6: Pulling fs layer
2022-12-01T10:03:09.3709229Z 7334631630ca: Pulling fs layer
2022-12-01T10:03:09.3709427Z b7010c1141c8: Pulling fs layer
2022-12-01T10:03:09.3709610Z 1d69ebcc1824: Pulling fs layer
2022-12-01T10:03:09.3709891Z bbd83461fb41: Pulling fs layer
2022-12-01T10:03:09.3710192Z bcd5197bc3b1: Pulling fs layer
2022-12-01T10:03:09.3710489Z b80f27646ab6: Pulling fs layer
2022-12-01T10:03:09.3710907Z 63ae5eb88541: Waiting
2022-12-01T10:03:09.3711230Z dce0658383e3: Waiting
2022-12-01T10:03:09.3711695Z 866c2acc4910: Pulling fs layer
2022-12-01T10:03:09.3711969Z 38c05cfe4070: Waiting
2022-12-01T10:03:09.3712180Z e39e0e87bf58: Pulling fs layer
2022-12-01T10:03:09.3712493Z fda59f894d4a: Waiting
2022-12-01T10:03:09.3712935Z 31e492142d52: Pulling fs layer
2022-12-01T10:03:09.3713301Z 7d95e9c9554c: Pulling fs layer
2022-12-01T10:03:09.3713514Z c4cd62aabb9c: Waiting
2022-12-01T10:03:09.3713682Z 17be6d40a7e6: Waiting
2022-12-01T10:03:09.3713867Z f405cc5c9c77: Pulling fs layer
2022-12-01T10:03:09.3714052Z d84da0b3b529: Waiting
2022-12-01T10:03:09.3714212Z d1d5291b2ab5: Waiting
2022-12-01T10:03:09.3714396Z a1d89816b283: Pulling fs layer
2022-12-01T10:03:09.3714577Z 81e2ac298c61: Waiting
2022-12-01T10:03:09.3714731Z 7334631630ca: Waiting
2022-12-01T10:03:09.3714912Z 38d7b060d3d4: Pulling fs layer
2022-12-01T10:03:09.3715094Z b7010c1141c8: Waiting
2022-12-01T10:03:09.3715259Z 0b48564513b3: Pulling fs layer
2022-12-01T10:03:09.3715439Z 866c2acc4910: Waiting
2022-12-01T10:03:09.3715612Z b80f27646ab6: Waiting
2022-12-01T10:03:09.3715776Z c517839d268c: Pulling fs layer
2022-12-01T10:03:09.3715955Z 1d69ebcc1824: Waiting
2022-12-01T10:03:09.3716134Z 9200027c6378: Pulling fs layer
2022-12-01T10:03:09.3716305Z e39e0e87bf58: Waiting
2022-12-01T10:03:09.3716477Z bcd5197bc3b1: Waiting
2022-12-01T10:03:09.3716721Z 1a68c742e877: Pulling fs layer
2022-12-01T10:03:09.3716902Z 085aceb64709: Pulling fs layer
2022-12-01T10:03:09.3717087Z bbd83461fb41: Waiting
2022-12-01T10:03:09.3717264Z a1d89816b283: Waiting
2022-12-01T10:03:09.3717534Z 31e492142d52: Waiting
2022-12-01T10:03:09.3717726Z 9fad26f5e1f4: Pulling fs layer
2022-12-01T10:03:09.3717944Z 7d95e9c9554c: Waiting
2022-12-01T10:03:09.3721930Z 3454ac6b1f68: Pulling fs layer
2022-12-01T10:03:09.3722261Z f405cc5c9c77: Waiting
2022-12-01T10:03:09.3722542Z 38d7b060d3d4: Waiting
2022-12-01T10:03:09.3722833Z 38599e0ca649: Pulling fs layer
2022-12-01T10:03:09.3723124Z 0b48564513b3: Waiting
2022-12-01T10:03:09.3723470Z a16f0b09d742: Pulling fs layer
2022-12-01T10:03:09.3723812Z 2b6269e5f012: Waiting
2022-12-01T10:03:09.3724079Z 100cd65220ba: Pulling fs layer
2022-12-01T10:03:09.3724296Z 085aceb64709: Waiting
2022-12-01T10:03:09.3724643Z 1a68c742e877: Waiting
2022-12-01T10:03:09.3725040Z 9d7685da3dfa: Pulling fs layer
2022-12-01T10:03:09.3725416Z 49405b81ba46: Waiting
2022-12-01T10:03:09.3725744Z 9200027c6378: Waiting
2022-12-01T10:03:09.3726127Z e69408c62fb4: Pulling fs layer
2022-12-01T10:03:09.3726485Z 1530f151c058: Pulling fs layer
2022-12-01T10:03:09.3726818Z 9fad26f5e1f4: Waiting
2022-12-01T10:03:09.3727178Z f5bd51a8525a: Waiting
2022-12-01T10:03:09.3727525Z a16f0b09d742: Waiting
2022-12-01T10:03:09.3727929Z 3aeaec05c672: Pulling fs layer
2022-12-01T10:03:09.3728380Z 99763bc3448a: Pulling fs layer
2022-12-01T10:03:09.3728741Z 1530f151c058: Waiting
2022-12-01T10:03:09.3729132Z cec809ad8b3c: Pulling fs layer
2022-12-01T10:03:09.3729548Z 677fdb68d2c6: Pulling fs layer
2022-12-01T10:03:09.3729900Z e69408c62fb4: Waiting
2022-12-01T10:03:09.3730294Z 92d94c0f43b4: Pulling fs layer
2022-12-01T10:03:09.3730689Z 100cd65220ba: Waiting
2022-12-01T10:03:09.3731028Z 99763bc3448a: Waiting
2022-12-01T10:03:09.3731388Z cec809ad8b3c: Waiting
2022-12-01T10:03:09.3731782Z a90bc9ac3f47: Pulling fs layer
2022-12-01T10:03:09.3732182Z 8ef4cf23833b: Pulling fs layer
2022-12-01T10:03:09.3732575Z 9d7685da3dfa: Waiting
2022-12-01T10:03:09.3732939Z 49809737e89d: Waiting
2022-12-01T10:03:09.3741422Z a90bc9ac3f47: Waiting
2022-12-01T10:03:09.3741694Z 92d94c0f43b4: Waiting
2022-12-01T10:03:09.3741872Z 3aeaec05c672: Waiting
2022-12-01T10:03:09.4442311Z 3a31ffa0162f: Verifying Checksum
2022-12-01T10:03:09.4442717Z 3a31ffa0162f: Download complete
2022-12-01T10:03:09.5225204Z 422bea31bd2a: Download complete
2022-12-01T10:03:09.5993294Z 27ad548906a1: Verifying Checksum
2022-12-01T10:03:09.5993659Z 27ad548906a1: Download complete
2022-12-01T10:03:09.6695490Z efe762c046e3: Verifying Checksum
2022-12-01T10:03:09.6695752Z efe762c046e3: Download complete
2022-12-01T10:03:09.6978178Z 675920708c8b: Verifying Checksum
2022-12-01T10:03:09.6978587Z 675920708c8b: Download complete
2022-12-01T10:03:09.7517565Z a12af00c9224: Verifying Checksum
2022-12-01T10:03:09.7517956Z a12af00c9224: Download complete
2022-12-01T10:03:09.7679662Z 49809737e89d: Verifying Checksum
2022-12-01T10:03:09.7680112Z 49809737e89d: Download complete
2022-12-01T10:03:09.8302139Z 4bd15b741963: Download complete
2022-12-01T10:03:09.8673427Z 9fdf20fee962: Verifying Checksum
2022-12-01T10:03:09.8673731Z 9fdf20fee962: Download complete
2022-12-01T10:03:09.9397635Z e3608d8edfea: Download complete
2022-12-01T10:03:10.0144839Z 63ae5eb88541: Verifying Checksum
2022-12-01T10:03:10.0145201Z 63ae5eb88541: Download complete
2022-12-01T10:03:10.4357083Z 675920708c8b: Pull complete
2022-12-01T10:03:10.5577601Z 3a31ffa0162f: Pull complete
2022-12-01T10:03:10.7021336Z 85d2fa9daf54: Verifying Checksum
2022-12-01T10:03:10.7021641Z 85d2fa9daf54: Download complete
2022-12-01T10:03:10.7840064Z 49405b81ba46: Download complete
2022-12-01T10:03:10.8768417Z f5bd51a8525a: Verifying Checksum
2022-12-01T10:03:10.8768795Z f5bd51a8525a: Download complete
2022-12-01T10:03:11.3625174Z dce0658383e3: Verifying Checksum
2022-12-01T10:03:11.3625548Z dce0658383e3: Download complete
2022-12-01T10:03:11.4247279Z fda59f894d4a: Download complete
2022-12-01T10:03:11.5242474Z c4cd62aabb9c: Download complete
2022-12-01T10:03:11.6152886Z d84da0b3b529: Verifying Checksum
2022-12-01T10:03:11.6153322Z d84da0b3b529: Download complete
2022-12-01T10:03:11.6983045Z 81e2ac298c61: Verifying Checksum
2022-12-01T10:03:11.6983346Z 81e2ac298c61: Download complete
2022-12-01T10:03:11.7720188Z 38c05cfe4070: Verifying Checksum
2022-12-01T10:03:11.7720648Z 38c05cfe4070: Download complete
2022-12-01T10:03:11.8639499Z af4d19351fdf: Verifying Checksum
2022-12-01T10:03:11.8639900Z af4d19351fdf: Download complete
2022-12-01T10:03:11.9451519Z d1d5291b2ab5: Verifying Checksum
2022-12-01T10:03:12.4806892Z d1d5291b2ab5: Download complete
2022-12-01T10:03:12.4807291Z fe9b24415086: Verifying Checksum
2022-12-01T10:03:12.6092300Z fe9b24415086: Download complete
2022-12-01T10:03:12.6092626Z 7334631630ca: Verifying Checksum
2022-12-01T10:03:12.6092818Z 7334631630ca: Download complete
2022-12-01T10:03:12.7156381Z b7010c1141c8: Verifying Checksum
2022-12-01T10:03:12.7156856Z b7010c1141c8: Download complete
2022-12-01T10:03:12.8126352Z 1d69ebcc1824: Verifying Checksum
2022-12-01T10:03:12.8126738Z 1d69ebcc1824: Download complete
2022-12-01T10:03:12.8810705Z bbd83461fb41: Download complete
2022-12-01T10:03:12.8873864Z 17be6d40a7e6: Verifying Checksum
2022-12-01T10:03:12.8874373Z 17be6d40a7e6: Download complete
2022-12-01T10:03:12.9653921Z bcd5197bc3b1: Download complete
2022-12-01T10:03:13.0365579Z 866c2acc4910: Verifying Checksum
2022-12-01T10:03:13.0365949Z 866c2acc4910: Download complete
2022-12-01T10:03:13.1560400Z e39e0e87bf58: Download complete
2022-12-01T10:03:13.2448210Z 31e492142d52: Verifying Checksum
2022-12-01T10:03:13.2448603Z 31e492142d52: Download complete
2022-12-01T10:03:13.3312636Z 7d95e9c9554c: Download complete
2022-12-01T10:03:13.4068367Z f405cc5c9c77: Verifying Checksum
2022-12-01T10:03:13.4068883Z f405cc5c9c77: Download complete
2022-12-01T10:03:13.4756947Z a1d89816b283: Verifying Checksum
2022-12-01T10:03:13.4757296Z a1d89816b283: Download complete
2022-12-01T10:03:13.5464005Z 38d7b060d3d4: Verifying Checksum
2022-12-01T10:03:13.5464416Z 38d7b060d3d4: Download complete
2022-12-01T10:03:13.6174356Z 0b48564513b3: Verifying Checksum
2022-12-01T10:03:13.6174614Z 0b48564513b3: Download complete
2022-12-01T10:03:13.7085923Z c517839d268c: Verifying Checksum
2022-12-01T10:03:13.7086313Z c517839d268c: Download complete
2022-12-01T10:03:13.7834263Z 9200027c6378: Verifying Checksum
2022-12-01T10:03:13.7834737Z 9200027c6378: Download complete
2022-12-01T10:03:13.8480876Z 1a68c742e877: Download complete
2022-12-01T10:03:13.9315609Z 085aceb64709: Download complete
2022-12-01T10:03:14.3991762Z 9fad26f5e1f4: Verifying Checksum
2022-12-01T10:03:14.3992065Z 9fad26f5e1f4: Download complete
2022-12-01T10:03:14.4851733Z 3454ac6b1f68: Verifying Checksum
2022-12-01T10:03:14.4852170Z 3454ac6b1f68: Download complete
2022-12-01T10:03:14.5508846Z 38599e0ca649: Download complete
2022-12-01T10:03:14.6409722Z a16f0b09d742: Download complete
2022-12-01T10:03:14.7226445Z 100cd65220ba: Download complete
2022-12-01T10:03:14.9763459Z 9d7685da3dfa: Verifying Checksum
2022-12-01T10:03:14.9763838Z 9d7685da3dfa: Download complete
2022-12-01T10:03:15.0573828Z e69408c62fb4: Verifying Checksum
2022-12-01T10:03:15.0574774Z e69408c62fb4: Download complete
2022-12-01T10:03:15.1298724Z b80f27646ab6: Verifying Checksum
2022-12-01T10:03:15.1299025Z b80f27646ab6: Download complete
2022-12-01T10:03:15.1389531Z 1530f151c058: Verifying Checksum
2022-12-01T10:03:15.1389940Z 1530f151c058: Download complete
2022-12-01T10:03:15.2359723Z 99763bc3448a: Verifying Checksum
2022-12-01T10:03:15.2360162Z 99763bc3448a: Download complete
2022-12-01T10:03:15.3268504Z cec809ad8b3c: Download complete
2022-12-01T10:03:15.4541027Z 677fdb68d2c6: Download complete
2022-12-01T10:03:15.5578549Z 92d94c0f43b4: Verifying Checksum
2022-12-01T10:03:15.5579013Z 92d94c0f43b4: Download complete
2022-12-01T10:03:15.6377489Z a90bc9ac3f47: Verifying Checksum
2022-12-01T10:03:15.6377886Z a90bc9ac3f47: Download complete
2022-12-01T10:03:16.2428636Z 8ef4cf23833b: Verifying Checksum
2022-12-01T10:03:16.2429090Z 8ef4cf23833b: Download complete
2022-12-01T10:03:18.3707466Z 3aeaec05c672: Verifying Checksum
2022-12-01T10:03:18.3707741Z 3aeaec05c672: Download complete
2022-12-01T10:03:19.1244049Z fe9b24415086: Pull complete
2022-12-01T10:03:19.3284948Z 422bea31bd2a: Pull complete
2022-12-01T10:03:19.5224543Z 27ad548906a1: Pull complete
2022-12-01T10:03:19.7469613Z efe762c046e3: Pull complete
2022-12-01T10:03:19.9801143Z a12af00c9224: Pull complete
2022-12-01T10:03:20.2327653Z 49809737e89d: Pull complete
2022-12-01T10:03:20.5007256Z 4bd15b741963: Pull complete
2022-12-01T10:03:20.7465021Z 9fdf20fee962: Pull complete
2022-12-01T10:03:21.9678544Z 2b6269e5f012: Verifying Checksum
2022-12-01T10:03:21.9678915Z 2b6269e5f012: Download complete
2022-12-01T10:03:22.5729479Z 85d2fa9daf54: Pull complete
2022-12-01T10:03:22.8003328Z e3608d8edfea: Pull complete
2022-12-01T10:03:23.0393646Z 63ae5eb88541: Pull complete
2022-12-01T10:03:42.9371463Z 2b6269e5f012: Pull complete
2022-12-01T10:03:43.3591887Z 49405b81ba46: Pull complete
2022-12-01T10:03:43.6728930Z f5bd51a8525a: Pull complete
2022-12-01T10:03:44.4782765Z dce0658383e3: Pull complete
2022-12-01T10:03:44.5858440Z fda59f894d4a: Pull complete
2022-12-01T10:03:44.7934101Z c4cd62aabb9c: Pull complete
2022-12-01T10:03:44.9547196Z d84da0b3b529: Pull complete
2022-12-01T10:03:45.0846009Z 81e2ac298c61: Pull complete
2022-12-01T10:03:45.2097121Z 38c05cfe4070: Pull complete
2022-12-01T10:03:45.3179427Z af4d19351fdf: Pull complete
2022-12-01T10:03:45.4172015Z d1d5291b2ab5: Pull complete
2022-12-01T10:03:47.2591303Z 17be6d40a7e6: Pull complete
2022-12-01T10:03:47.4687496Z 7334631630ca: Pull complete
2022-12-01T10:03:47.5964810Z b7010c1141c8: Pull complete
2022-12-01T10:03:47.8284552Z 1d69ebcc1824: Pull complete
2022-12-01T10:03:48.0818334Z bbd83461fb41: Pull complete
2022-12-01T10:03:48.3164149Z bcd5197bc3b1: Pull complete
2022-12-01T10:03:52.9229587Z b80f27646ab6: Pull complete
2022-12-01T10:03:53.1490188Z 866c2acc4910: Pull complete
2022-12-01T10:03:53.3497313Z e39e0e87bf58: Pull complete
2022-12-01T10:03:53.5469051Z 31e492142d52: Pull complete
2022-12-01T10:03:53.6851974Z 7d95e9c9554c: Pull complete
2022-12-01T10:03:53.7755937Z f405cc5c9c77: Pull complete
2022-12-01T10:03:53.8836361Z a1d89816b283: Pull complete
2022-12-01T10:03:53.9932221Z 38d7b060d3d4: Pull complete
2022-12-01T10:03:54.0980133Z 0b48564513b3: Pull complete
2022-12-01T10:03:54.1939925Z c517839d268c: Pull complete
2022-12-01T10:03:54.2976951Z 9200027c6378: Pull complete
2022-12-01T10:03:54.3773064Z 1a68c742e877: Pull complete
2022-12-01T10:03:54.4824091Z 085aceb64709: Pull complete
2022-12-01T10:03:55.7194043Z 9fad26f5e1f4: Pull complete
2022-12-01T10:03:55.9141321Z 3454ac6b1f68: Pull complete
2022-12-01T10:03:56.1856442Z 38599e0ca649: Pull complete
2022-12-01T10:03:56.4424169Z a16f0b09d742: Pull complete
2022-12-01T10:03:56.6518276Z 100cd65220ba: Pull complete
2022-12-01T10:03:57.0943993Z 9d7685da3dfa: Pull complete
2022-12-01T10:03:57.3304571Z e69408c62fb4: Pull complete
2022-12-01T10:03:57.5521901Z 1530f151c058: Pull complete
2022-12-01T10:04:01.5454169Z 3aeaec05c672: Pull complete
2022-12-01T10:04:01.7639732Z 99763bc3448a: Pull complete
2022-12-01T10:04:01.9990840Z cec809ad8b3c: Pull complete
2022-12-01T10:04:02.2314748Z 677fdb68d2c6: Pull complete
2022-12-01T10:04:02.4631270Z 92d94c0f43b4: Pull complete
2022-12-01T10:04:02.6807612Z a90bc9ac3f47: Pull complete
2022-12-01T10:04:04.3045342Z 8ef4cf23833b: Pull complete
2022-12-01T10:04:04.4253711Z Digest: sha256:7d4daa1530ad815417dd92428a12643a4a8b0e93e820e2eb2d4e8818a4e57ba2
2022-12-01T10:04:04.4691341Z Status: Downloaded newer image for 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-py3.7-gcc7:fa72f5a0a230eb632055220542038bd4ceca184b
2022-12-01T10:04:04.4933432Z 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-py3.7-gcc7:fa72f5a0a230eb632055220542038bd4ceca184b
2022-12-01T10:04:04.4986611Z ##[group]Run python3 -m pip install psutil==5.9.1
2022-12-01T10:04:04.4986977Z python3 -m pip install psutil==5.9.1
2022-12-01T10:04:04.4987284Z python3 -m pip install pynvml==11.4.1
2022-12-01T10:04:04.4987588Z python3 -m tools.stats.monitor > usage_log.txt 2>&1 &
2022-12-01T10:04:04.4987927Z echo "::set-output name=monitor-script-pid::${!}"
2022-12-01T10:04:04.4999294Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-12-01T10:04:04.4999619Z env:
2022-12-01T10:04:04.4999815Z   GIT_DEFAULT_BRANCH: master
2022-12-01T10:04:04.5000104Z ##[endgroup]
2022-12-01T10:04:04.7172528Z Defaulting to user installation because normal site-packages is not writeable
2022-12-01T10:04:04.7358077Z Requirement already satisfied: psutil==5.9.1 in /home/ec2-user/.local/lib/python3.7/site-packages (5.9.1)
2022-12-01T10:04:05.1647667Z Defaulting to user installation because normal site-packages is not writeable
2022-12-01T10:04:05.1831210Z Requirement already satisfied: pynvml==11.4.1 in /home/ec2-user/.local/lib/python3.7/site-packages (11.4.1)
2022-12-01T10:04:05.3959431Z ##[warning]The `set-output` command is deprecated and will be disabled soon. Please upgrade to using Environment Files. For more information see: https://github.blog/changelog/2022-10-11-github-actions-deprecating-save-state-and-set-output-commands/
2022-12-01T10:04:05.4000596Z Prepare all required actions
2022-12-01T10:04:05.4000865Z Getting action download info
2022-12-01T10:04:05.6005634Z Download action repository 'seemethere/download-artifact-s3@v4' (SHA:4a8bfae15cc25cc0785c1603ee87a9da8fd442ea)
2022-12-01T10:04:05.7761813Z Download action repository 'actions/download-artifact@v2' (SHA:f023be2c48cc18debc3bacd34cb396e0295e2869)
2022-12-01T10:04:05.9171525Z ##[group]Run ./.github/actions/download-build-artifacts
2022-12-01T10:04:05.9171766Z with:
2022-12-01T10:04:05.9171951Z   name: linux-focal-py3.7-gcc7
2022-12-01T10:04:05.9172147Z env:
2022-12-01T10:04:05.9172317Z   GIT_DEFAULT_BRANCH: master
2022-12-01T10:04:05.9172503Z ##[endgroup]
2022-12-01T10:04:05.9197133Z ##[group]Run seemethere/download-artifact-s3@v4
2022-12-01T10:04:05.9197364Z with:
2022-12-01T10:04:05.9197540Z   name: linux-focal-py3.7-gcc7
2022-12-01T10:04:05.9197749Z   s3-bucket: gha-artifacts
2022-12-01T10:04:05.9197940Z   region: us-east-1
2022-12-01T10:04:05.9198096Z env:
2022-12-01T10:04:05.9198299Z   GIT_DEFAULT_BRANCH: master
2022-12-01T10:04:05.9198473Z ##[endgroup]
2022-12-01T10:04:06.3501861Z Found 1 objects with prefix pytorch/pytorch/3591403534/linux-focal-py3.7-gcc7/
2022-12-01T10:04:06.3502662Z Starting download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip
2022-12-01T10:04:09.3818100Z Finished download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip
2022-12-01T10:04:09.3818384Z 
2022-12-01T10:04:09.3821890Z ##[warning]The `set-output` command is deprecated and will be disabled soon. Please upgrade to using Environment Files. For more information see: https://github.blog/changelog/2022-10-11-github-actions-deprecating-save-state-and-set-output-commands/
2022-12-01T10:04:09.3823369Z Artifact download has finished successfully
2022-12-01T10:04:09.3934775Z ##[group]Run unzip -o artifacts.zip
2022-12-01T10:04:09.3935013Z unzip -o artifacts.zip
2022-12-01T10:04:09.3946848Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-12-01T10:04:09.3947060Z env:
2022-12-01T10:04:09.3947236Z   GIT_DEFAULT_BRANCH: master
2022-12-01T10:04:09.3947424Z ##[endgroup]
2022-12-01T10:04:09.3982423Z Archive:  artifacts.zip
2022-12-01T10:04:09.3984044Z    creating: dist/
2022-12-01T10:04:10.1673917Z   inflating: dist/torch-1.13.0a0+gitc13d400-cp37-cp37m-linux_x86_64.whl  
2022-12-01T10:04:10.1674382Z    creating: build/custom_test_artifacts/
2022-12-01T10:04:10.1674691Z    creating: build/custom_test_artifacts/custom-op-build/
2022-12-01T10:04:10.1675074Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/
2022-12-01T10:04:10.1677090Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeOutput.log  
2022-12-01T10:04:10.1677645Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/
2022-12-01T10:04:10.1678071Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CMakeSystem.cmake  
2022-12-01T10:04:10.1678505Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdC/
2022-12-01T10:04:10.1678911Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdC/tmp/
2022-12-01T10:04:10.1679837Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdC/CMakeCCompilerId.c  
2022-12-01T10:04:10.1681297Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdC/a.out  
2022-12-01T10:04:10.1681725Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCXX/
2022-12-01T10:04:10.1682146Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCXX/tmp/
2022-12-01T10:04:10.1683689Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCXX/CMakeCXXCompilerId.cpp  
2022-12-01T10:04:10.1684978Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCXX/a.out  
2022-12-01T10:04:10.1686523Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_C.bin  
2022-12-01T10:04:10.1686985Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CMakeCCompiler.cmake  
2022-12-01T10:04:10.1688259Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_CXX.bin  
2022-12-01T10:04:10.1688873Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CMakeCXXCompiler.cmake  
2022-12-01T10:04:10.1689292Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeTmp/
2022-12-01T10:04:10.1689699Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeError.log  
2022-12-01T10:04:10.1690098Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/cmake.check_cache  
2022-12-01T10:04:10.1690508Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/
2022-12-01T10:04:10.1690956Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.ts  
2022-12-01T10:04:10.1691426Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.make  
2022-12-01T10:04:10.1691864Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/depend.make  
2022-12-01T10:04:10.1692303Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/link.txt  
2022-12-01T10:04:10.1692749Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/cmake_clean.cmake  
2022-12-01T10:04:10.1693294Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/build.make  
2022-12-01T10:04:10.1693733Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/DependInfo.cmake  
2022-12-01T10:04:10.1694171Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/flags.make  
2022-12-01T10:04:10.1694615Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/progress.make  
2022-12-01T10:04:10.1709939Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o.d  
2022-12-01T10:04:10.1797209Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o  
2022-12-01T10:04:10.1797676Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/
2022-12-01T10:04:10.1798138Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.ts  
2022-12-01T10:04:10.1798630Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.make  
2022-12-01T10:04:10.1799089Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/depend.make  
2022-12-01T10:04:10.1799543Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/link.txt  
2022-12-01T10:04:10.1800012Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/cmake_clean.cmake  
2022-12-01T10:04:10.1800472Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/build.make  
2022-12-01T10:04:10.1800913Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/DependInfo.cmake  
2022-12-01T10:04:10.1801370Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/flags.make  
2022-12-01T10:04:10.1801826Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/progress.make  
2022-12-01T10:04:10.1817034Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o.d  
2022-12-01T10:04:10.1880003Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o  
2022-12-01T10:04:10.1880501Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeDirectoryInformation.cmake  
2022-12-01T10:04:10.1881160Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/TargetDirectories.txt  
2022-12-01T10:04:10.1881592Z  extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/progress.marks  
2022-12-01T10:04:10.1881988Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile2  
2022-12-01T10:04:10.1882394Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile.cmake  
2022-12-01T10:04:10.1883689Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeCache.txt  
2022-12-01T10:04:10.1884509Z   inflating: build/custom_test_artifacts/custom-op-build/Makefile  
2022-12-01T10:04:10.1884972Z   inflating: build/custom_test_artifacts/custom-op-build/cmake_install.cmake  
2022-12-01T10:04:10.1956542Z   inflating: build/custom_test_artifacts/custom-op-build/libcustom_ops.so  
2022-12-01T10:04:10.2004393Z   inflating: build/custom_test_artifacts/custom-op-build/test_custom_ops  
2022-12-01T10:04:10.2005053Z    creating: build/custom_test_artifacts/jit-hook-build/
2022-12-01T10:04:10.2005434Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/
2022-12-01T10:04:10.2007389Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeOutput.log  
2022-12-01T10:04:10.2008073Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/
2022-12-01T10:04:10.2008822Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CMakeSystem.cmake  
2022-12-01T10:04:10.2009256Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdC/
2022-12-01T10:04:10.2009868Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdC/tmp/
2022-12-01T10:04:10.2010765Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdC/CMakeCCompilerId.c  
2022-12-01T10:04:10.2011725Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdC/a.out  
2022-12-01T10:04:10.2012483Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCXX/
2022-12-01T10:04:10.2012952Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCXX/tmp/
2022-12-01T10:04:10.2014257Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCXX/CMakeCXXCompilerId.cpp  
2022-12-01T10:04:10.2015493Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCXX/a.out  
2022-12-01T10:04:10.2017057Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_C.bin  
2022-12-01T10:04:10.2017873Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CMakeCCompiler.cmake  
2022-12-01T10:04:10.2018817Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_CXX.bin  
2022-12-01T10:04:10.2019750Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CMakeCXXCompiler.cmake  
2022-12-01T10:04:10.2020630Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeTmp/
2022-12-01T10:04:10.2021359Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeError.log  
2022-12-01T10:04:10.2022089Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/cmake.check_cache  
2022-12-01T10:04:10.2022818Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/
2022-12-01T10:04:10.2023623Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.ts  
2022-12-01T10:04:10.2024462Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.make  
2022-12-01T10:04:10.2025311Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/depend.make  
2022-12-01T10:04:10.2025776Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/link.txt  
2022-12-01T10:04:10.2026207Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/cmake_clean.cmake  
2022-12-01T10:04:10.2026778Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/build.make  
2022-12-01T10:04:10.2027235Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/DependInfo.cmake  
2022-12-01T10:04:10.2027677Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/flags.make  
2022-12-01T10:04:10.2028111Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/progress.make  
2022-12-01T10:04:10.2040833Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o.d  
2022-12-01T10:04:10.2088983Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o  
2022-12-01T10:04:10.2089837Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeDirectoryInformation.cmake  
2022-12-01T10:04:10.2090647Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/TargetDirectories.txt  
2022-12-01T10:04:10.2091407Z  extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/progress.marks  
2022-12-01T10:04:10.2092034Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile2  
2022-12-01T10:04:10.2092567Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile.cmake  
2022-12-01T10:04:10.2093285Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeCache.txt  
2022-12-01T10:04:10.2093949Z   inflating: build/custom_test_artifacts/jit-hook-build/Makefile  
2022-12-01T10:04:10.2094480Z   inflating: build/custom_test_artifacts/jit-hook-build/cmake_install.cmake  
2022-12-01T10:04:10.2131466Z   inflating: build/custom_test_artifacts/jit-hook-build/test_jit_hooks  
2022-12-01T10:04:10.2132125Z    creating: build/custom_test_artifacts/custom-backend-build/
2022-12-01T10:04:10.2132507Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/
2022-12-01T10:04:10.2134539Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeOutput.log  
2022-12-01T10:04:10.2135283Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/
2022-12-01T10:04:10.2136019Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CMakeSystem.cmake  
2022-12-01T10:04:10.2136455Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdC/
2022-12-01T10:04:10.2136891Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdC/tmp/
2022-12-01T10:04:10.2137696Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdC/CMakeCCompilerId.c  
2022-12-01T10:04:10.2139021Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdC/a.out  
2022-12-01T10:04:10.2139893Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCXX/
2022-12-01T10:04:10.2140501Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCXX/tmp/
2022-12-01T10:04:10.2141769Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCXX/CMakeCXXCompilerId.cpp  
2022-12-01T10:04:10.2143018Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCXX/a.out  
2022-12-01T10:04:10.2144393Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_C.bin  
2022-12-01T10:04:10.2145256Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CMakeCCompiler.cmake  
2022-12-01T10:04:10.2146272Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_CXX.bin  
2022-12-01T10:04:10.2147152Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CMakeCXXCompiler.cmake  
2022-12-01T10:04:10.2147892Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeTmp/
2022-12-01T10:04:10.2148800Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeError.log  
2022-12-01T10:04:10.2149554Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/cmake.check_cache  
2022-12-01T10:04:10.2150351Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/
2022-12-01T10:04:10.2151188Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.ts  
2022-12-01T10:04:10.2152108Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.make  
2022-12-01T10:04:10.2152740Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/depend.make  
2022-12-01T10:04:10.2153198Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/link.txt  
2022-12-01T10:04:10.2153668Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/cmake_clean.cmake  
2022-12-01T10:04:10.2154140Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/build.make  
2022-12-01T10:04:10.2154601Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/DependInfo.cmake  
2022-12-01T10:04:10.2155218Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/flags.make  
2022-12-01T10:04:10.2155983Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/progress.make  
2022-12-01T10:04:10.2156562Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o.d  
2022-12-01T10:04:10.2270753Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o  
2022-12-01T10:04:10.2271645Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/
2022-12-01T10:04:10.2272541Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.ts  
2022-12-01T10:04:10.2273417Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.make  
2022-12-01T10:04:10.2274303Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/depend.make  
2022-12-01T10:04:10.2275165Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/link.txt  
2022-12-01T10:04:10.2275655Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/cmake_clean.cmake  
2022-12-01T10:04:10.2276125Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/build.make  
2022-12-01T10:04:10.2276597Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/DependInfo.cmake  
2022-12-01T10:04:10.2277085Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/flags.make  
2022-12-01T10:04:10.2277570Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/progress.make  
2022-12-01T10:04:10.2290531Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o.d  
2022-12-01T10:04:10.2335142Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o  
2022-12-01T10:04:10.2336069Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeDirectoryInformation.cmake  
2022-12-01T10:04:10.2336951Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/TargetDirectories.txt  
2022-12-01T10:04:10.2337754Z  extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/progress.marks  
2022-12-01T10:04:10.2338363Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile2  
2022-12-01T10:04:10.2339164Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile.cmake  
2022-12-01T10:04:10.2339988Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeCache.txt  
2022-12-01T10:04:10.2340791Z   inflating: build/custom_test_artifacts/custom-backend-build/Makefile  
2022-12-01T10:04:10.2341179Z   inflating: build/custom_test_artifacts/custom-backend-build/cmake_install.cmake  
2022-12-01T10:04:10.2432108Z   inflating: build/custom_test_artifacts/custom-backend-build/libcustom_backend.so  
2022-12-01T10:04:10.2467871Z   inflating: build/custom_test_artifacts/custom-backend-build/test_custom_backend  
2022-12-01T10:04:10.2468366Z    creating: build/lib/
2022-12-01T10:04:10.2468899Z   inflating: build/lib/libclog.a     
2022-12-01T10:04:10.2519793Z   inflating: build/lib/libgtest.a    
2022-12-01T10:04:10.2527615Z   inflating: build/lib/libpthreadpool.a  
2022-12-01T10:04:10.2599148Z   inflating: build/lib/libbenchmark.a  
2022-12-01T10:04:10.2605771Z   inflating: build/lib/libittnotify.a  
2022-12-01T10:04:10.2688023Z   inflating: build/lib/libprotobuf-lite.a  
2022-12-01T10:04:10.2712848Z   inflating: build/lib/libtensorpipe_uv.a  
2022-12-01T10:04:10.2803117Z   inflating: build/lib/libgloo.a     
2022-12-01T10:04:10.2863134Z   inflating: build/lib/libasmjit.a   
2022-12-01T10:04:10.3277527Z   inflating: build/lib/libprotobuf.a  
2022-12-01T10:04:10.3292486Z   inflating: build/lib/libfmt.a      
2022-12-01T10:04:10.3292994Z   inflating: build/lib/libfoxi_loader.a  
2022-12-01T10:04:10.3352740Z   inflating: build/lib/libc10.so     
2022-12-01T10:04:10.3353929Z   inflating: build/lib/libtorch_global_deps.so  
2022-12-01T10:04:10.3361244Z   inflating: build/lib/libcpuinfo.a  
2022-12-01T10:04:10.3368325Z   inflating: build/lib/libcpuinfo_internals.a  
2022-12-01T10:04:10.3380782Z   inflating: build/lib/libqnnpack.a  
2022-12-01T10:04:10.3399413Z   inflating: build/lib/libpytorch_qnnpack.a  
2022-12-01T10:04:10.3401280Z   inflating: build/lib/libnnpack_reference_layers.a  
2022-12-01T10:04:10.3415748Z   inflating: build/lib/libgmock.a    
2022-12-01T10:04:10.3868678Z   inflating: build/lib/libprotoc.a   
2022-12-01T10:04:10.3869443Z   inflating: build/lib/libgtest_main.a  
2022-12-01T10:04:10.3869864Z   inflating: build/lib/libbenchmark_main.a  
2022-12-01T10:04:11.0201315Z   inflating: build/lib/libdnnl.a     
2022-12-01T10:04:11.0218681Z   inflating: build/lib/libnnpack.a   
2022-12-01T10:04:11.0738969Z   inflating: build/lib/libtensorpipe.a  
2022-12-01T10:04:11.1941007Z   inflating: build/lib/libfbgemm.a   
2022-12-01T10:04:11.2813404Z   inflating: build/lib/libdnnl_graph.a  
2022-12-01T10:04:11.2813805Z   inflating: build/lib/libgmock_main.a  
2022-12-01T10:04:11.3004829Z   inflating: build/lib/libkineto.a   
2022-12-01T10:04:11.3039859Z   inflating: build/lib/libcaffe2_protos.a  
2022-12-01T10:04:11.3076712Z   inflating: build/lib/libonnx_proto.a  
2022-12-01T10:04:11.3186903Z   inflating: build/lib/libXNNPACK.a  
2022-12-01T10:04:11.3712215Z   inflating: build/lib/libonnx.a     
2022-12-01T10:04:13.1856693Z   inflating: build/lib/libtorch_cpu.so  
2022-12-01T10:04:13.1857226Z   inflating: build/lib/libtorch.so   
2022-12-01T10:04:13.1876209Z   inflating: build/lib/libjitbackend_test.so  
2022-12-01T10:04:13.1923453Z   inflating: build/lib/libtorchbind_test.so  
2022-12-01T10:04:13.1948231Z   inflating: build/lib/libbackend_with_compiler.so  
2022-12-01T10:04:13.1951794Z   inflating: build/lib/libshm.so     
2022-12-01T10:04:13.3267208Z   inflating: build/lib/libtorch_python.so  
2022-12-01T10:04:13.3298250Z   inflating: build/lib/libnnapi_backend.so  
2022-12-01T10:04:13.3298480Z    creating: build/bin/
2022-12-01T10:04:13.3339895Z   inflating: build/bin/c10_CompileTimeFunctionPointer_test  
2022-12-01T10:04:13.3384100Z   inflating: build/bin/c10_DeviceGuard_test  
2022-12-01T10:04:13.3426292Z   inflating: build/bin/c10_Device_test  
2022-12-01T10:04:13.3476057Z   inflating: build/bin/c10_DispatchKeySet_test  
2022-12-01T10:04:13.3516563Z   inflating: build/bin/c10_StreamGuard_test  
2022-12-01T10:04:13.3559070Z   inflating: build/bin/c10_SymInt_test  
2022-12-01T10:04:13.3605882Z   inflating: build/bin/c10_InlineDeviceGuard_test  
2022-12-01T10:04:13.3653003Z   inflating: build/bin/c10_InlineStreamGuard_test  
2022-12-01T10:04:13.3693822Z   inflating: build/bin/c10_Array_test  
2022-12-01T10:04:13.3742468Z   inflating: build/bin/c10_SizesAndStrides_test  
2022-12-01T10:04:13.3787141Z   inflating: build/bin/c10_Bitset_test  
2022-12-01T10:04:13.3827786Z   inflating: build/bin/c10_ConstexprCrc_test  
2022-12-01T10:04:13.3870774Z   inflating: build/bin/c10_C++17_test  
2022-12-01T10:04:13.3912068Z   inflating: build/bin/c10_DeadlockDetection_test  
2022-12-01T10:04:13.3954167Z   inflating: build/bin/c10_Half_test  
2022-12-01T10:04:13.4002048Z   inflating: build/bin/c10_LeftRight_test  
2022-12-01T10:04:13.4054943Z   inflating: build/bin/c10_Metaprogramming_test  
2022-12-01T10:04:13.4097275Z   inflating: build/bin/c10_Synchronized_test  
2022-12-01T10:04:13.4218499Z   inflating: build/bin/c10_SmallVectorTest  
2022-12-01T10:04:13.4266952Z   inflating: build/bin/c10_ThreadLocal_test  
2022-12-01T10:04:13.4309385Z   inflating: build/bin/c10_TypeList_test  
2022-12-01T10:04:13.4353844Z   inflating: build/bin/c10_TypeIndex_test  
2022-12-01T10:04:13.4394362Z   inflating: build/bin/c10_TypeTraits_test  
2022-12-01T10:04:13.4438095Z   inflating: build/bin/c10_accumulate_test  
2022-12-01T10:04:13.4485110Z   inflating: build/bin/c10_bfloat16_test  
2022-12-01T10:04:13.4530275Z   inflating: build/bin/c10_complex_math_test  
2022-12-01T10:04:13.4577620Z   inflating: build/bin/c10_complex_test  
2022-12-01T10:04:13.4621737Z   inflating: build/bin/c10_exception_test  
2022-12-01T10:04:13.4714098Z   inflating: build/bin/c10_either_test  
2022-12-01T10:04:13.4756501Z   inflating: build/bin/c10_flags_test  
2022-12-01T10:04:13.4898998Z   inflating: build/bin/c10_intrusive_ptr_test  
2022-12-01T10:04:13.4941542Z   inflating: build/bin/c10_irange_test  
2022-12-01T10:04:13.4990481Z   inflating: build/bin/c10_logging_test  
2022-12-01T10:04:13.5053845Z   inflating: build/bin/c10_optional_test  
2022-12-01T10:04:13.5099923Z   inflating: build/bin/c10_registry_test  
2022-12-01T10:04:13.5152344Z   inflating: build/bin/c10_ordered_preserving_dict_test  
2022-12-01T10:04:13.5202067Z   inflating: build/bin/c10_string_view_test  
2022-12-01T10:04:13.5245316Z   inflating: build/bin/c10_tempfile_test  
2022-12-01T10:04:13.5292848Z   inflating: build/bin/c10_typeid_test  
2022-12-01T10:04:13.5340727Z   inflating: build/bin/c10_intrusive_ptr_benchmark  
2022-12-01T10:04:13.5752999Z   inflating: build/bin/protoc-3.13.0.0  
2022-12-01T10:04:13.6163205Z   inflating: build/bin/protoc        
2022-12-01T10:04:13.6410171Z   inflating: build/bin/vec_test_all_types_DEFAULT  
2022-12-01T10:04:13.6686003Z   inflating: build/bin/vec_test_all_types_AVX2  
2022-12-01T10:04:13.6731047Z   inflating: build/bin/FileStoreTest  
2022-12-01T10:04:13.6781899Z   inflating: build/bin/TCPStoreTest  
2022-12-01T10:04:13.6827311Z   inflating: build/bin/HashStoreTest  
2022-12-01T10:04:13.6829771Z   inflating: build/bin/example_allreduce  
2022-12-01T10:04:13.6891065Z   inflating: build/bin/Dict_test     
2022-12-01T10:04:13.6945894Z   inflating: build/bin/ProcessGroupGlooTest  
2022-12-01T10:04:13.6990092Z   inflating: build/bin/Dimname_test  
2022-12-01T10:04:13.7043439Z   inflating: build/bin/MaybeOwned_test  
2022-12-01T10:04:13.7091500Z   inflating: build/bin/NamedTensor_test  
2022-12-01T10:04:13.7141777Z   inflating: build/bin/apply_utils_test  
2022-12-01T10:04:13.7191283Z   inflating: build/bin/atest         
2022-12-01T10:04:13.7236976Z   inflating: build/bin/broadcast_test  
2022-12-01T10:04:13.7288540Z   inflating: build/bin/basic         
2022-12-01T10:04:13.7337908Z   inflating: build/bin/cpu_generator_test  
2022-12-01T10:04:13.7381975Z   inflating: build/bin/cpu_profiling_allocator_test  
2022-12-01T10:04:13.7456142Z   inflating: build/bin/cpu_rng_test  
2022-12-01T10:04:13.7497737Z   inflating: build/bin/dispatch_key_set_test  
2022-12-01T10:04:13.7539561Z   inflating: build/bin/dlconvertor_test  
2022-12-01T10:04:13.7588687Z   inflating: build/bin/extension_backend_test  
2022-12-01T10:04:13.7635572Z   inflating: build/bin/half_test     
2022-12-01T10:04:13.7715565Z   inflating: build/bin/ivalue_test   
2022-12-01T10:04:13.7757202Z   inflating: build/bin/lazy_tensor_test  
2022-12-01T10:04:13.7802096Z   inflating: build/bin/math_kernel_test  
2022-12-01T10:04:13.7847117Z   inflating: build/bin/memory_format_test  
2022-12-01T10:04:13.7891470Z   inflating: build/bin/memory_overlapping_test  
2022-12-01T10:04:13.7933959Z   inflating: build/bin/operator_name_test  
2022-12-01T10:04:13.7977811Z   inflating: build/bin/mobile_memory_cleanup  
2022-12-01T10:04:13.8025140Z   inflating: build/bin/native_test   
2022-12-01T10:04:13.8069277Z   inflating: build/bin/packedtensoraccessor_test  
2022-12-01T10:04:13.8111145Z   inflating: build/bin/operators_test  
2022-12-01T10:04:13.8166080Z   inflating: build/bin/pow_test      
2022-12-01T10:04:13.8214280Z   inflating: build/bin/quantized_test  
2022-12-01T10:04:13.8255896Z   inflating: build/bin/reduce_ops_test  
2022-12-01T10:04:13.8298534Z   inflating: build/bin/reportMemoryUsage_test  
2022-12-01T10:04:13.8345986Z   inflating: build/bin/scalar_tensor_test  
2022-12-01T10:04:13.8395411Z   inflating: build/bin/scalar_test   
2022-12-01T10:04:13.8439228Z   inflating: build/bin/stride_properties_test  
2022-12-01T10:04:13.8505914Z   inflating: build/bin/tensor_iterator_test  
2022-12-01T10:04:13.8507801Z   inflating: build/bin/thread_init_test  
2022-12-01T10:04:13.8554840Z   inflating: build/bin/type_ptr_test  
2022-12-01T10:04:13.8601712Z   inflating: build/bin/test_parallel  
2022-12-01T10:04:13.8643196Z   inflating: build/bin/variant_test  
2022-12-01T10:04:13.8686883Z   inflating: build/bin/undefined_tensor_test  
2022-12-01T10:04:13.8737529Z   inflating: build/bin/type_test     
2022-12-01T10:04:13.8738768Z   inflating: build/bin/verify_api_visibility  
2022-12-01T10:04:13.8797168Z   inflating: build/bin/vmap_test     
2022-12-01T10:04:13.8839823Z   inflating: build/bin/wrapdim_test  
2022-12-01T10:04:13.8882335Z   inflating: build/bin/weakref_test  
2022-12-01T10:04:13.8932663Z   inflating: build/bin/IListRef_test  
2022-12-01T10:04:13.9024767Z   inflating: build/bin/List_test     
2022-12-01T10:04:13.9065652Z   inflating: build/bin/xla_tensor_test  
2022-12-01T10:04:13.9168442Z   inflating: build/bin/kernel_function_legacy_test  
2022-12-01T10:04:13.9249637Z   inflating: build/bin/kernel_function_test  
2022-12-01T10:04:13.9304461Z   inflating: build/bin/KernelFunction_test  
2022-12-01T10:04:13.9412379Z   inflating: build/bin/kernel_lambda_legacy_test  
2022-12-01T10:04:13.9499562Z   inflating: build/bin/kernel_lambda_test  
2022-12-01T10:04:13.9549817Z   inflating: build/bin/kernel_stackbased_test  
2022-12-01T10:04:13.9592232Z   inflating: build/bin/CppSignature_test  
2022-12-01T10:04:13.9673129Z   inflating: build/bin/make_boxed_from_unboxed_functor_test  
2022-12-01T10:04:13.9713145Z   inflating: build/bin/op_allowlist_test  
2022-12-01T10:04:13.9757652Z   inflating: build/bin/inline_container_test  
2022-12-01T10:04:14.0000987Z   inflating: build/bin/op_registration_test  
2022-12-01T10:04:14.0048261Z   inflating: build/bin/backend_fallback_test  
2022-12-01T10:04:14.0062143Z   inflating: build/bin/tutorial_tensorexpr  
2022-12-01T10:04:14.0107595Z   inflating: build/bin/test_dist_autograd  
2022-12-01T10:04:14.0166595Z   inflating: build/bin/test_cpp_rpc  
2022-12-01T10:04:14.0168670Z   inflating: build/bin/parallel_benchmark  
2022-12-01T10:04:14.0227225Z   inflating: build/bin/test_mobile_nnc  
2022-12-01T10:04:14.0235869Z   inflating: build/bin/aot_model_compiler_test  
2022-12-01T10:04:14.0240180Z   inflating: build/bin/torch_shm_manager  
2022-12-01T10:04:14.0538298Z   inflating: build/bin/test_lazy     
2022-12-01T10:04:14.1222797Z   inflating: build/bin/test_tensorexpr  
2022-12-01T10:04:14.2237654Z   inflating: build/bin/test_api      
2022-12-01T10:04:14.2735400Z   inflating: build/bin/test_jit      
2022-12-01T10:04:14.2736397Z   inflating: .pytorch-test-times.json  
2022-12-01T10:04:14.2758890Z ##[group]Run df -H
2022-12-01T10:04:14.2759079Z df -H
2022-12-01T10:04:14.2770159Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-12-01T10:04:14.2770366Z env:
2022-12-01T10:04:14.2770546Z   GIT_DEFAULT_BRANCH: master
2022-12-01T10:04:14.2770735Z ##[endgroup]
2022-12-01T10:04:14.2802518Z Filesystem      Size  Used Avail Use% Mounted on
2022-12-01T10:04:14.2802894Z devtmpfs        8.2G     0  8.2G   0% /dev
2022-12-01T10:04:14.2803241Z tmpfs           8.2G   99k  8.2G   1% /dev/shm
2022-12-01T10:04:14.2803539Z tmpfs           8.2G  431k  8.2G   1% /run
2022-12-01T10:04:14.2803883Z tmpfs           8.2G     0  8.2G   0% /sys/fs/cgroup
2022-12-01T10:04:14.2804210Z /dev/nvme0n1p1  162G   16G  146G  10% /
2022-12-01T10:04:14.2907497Z ##[group]Run .github/scripts/parse_ref.py
2022-12-01T10:04:14.2907774Z .github/scripts/parse_ref.py
2022-12-01T10:04:14.2918861Z shell: /usr/bin/bash -e {0}
2022-12-01T10:04:14.2919045Z env:
2022-12-01T10:04:14.2919234Z   GIT_DEFAULT_BRANCH: master
2022-12-01T10:04:14.2919426Z ##[endgroup]
2022-12-01T10:04:14.3096565Z ##[warning]The `set-output` command is deprecated and will be disabled soon. Please upgrade to using Environment Files. For more information see: https://github.blog/changelog/2022-10-11-github-actions-deprecating-save-state-and-set-output-commands/
2022-12-01T10:04:14.3220597Z ##[group]Run set -x
2022-12-01T10:04:14.3220954Z set -x
2022-12-01T10:04:14.3221125Z 
2022-12-01T10:04:14.3221309Z if [[ $TEST_CONFIG == 'multigpu' ]]; then
2022-12-01T10:04:14.3221568Z   TEST_COMMAND=.jenkins/pytorch/multigpu-test.sh
2022-12-01T10:04:14.3221830Z elif [[ $BUILD_ENVIRONMENT == *onnx* ]]; then
2022-12-01T10:04:14.3222058Z   TEST_COMMAND=.jenkins/caffe2/test.sh
2022-12-01T10:04:14.3222260Z else
2022-12-01T10:04:14.3222463Z   TEST_COMMAND=.jenkins/pytorch/test.sh
2022-12-01T10:04:14.3222650Z fi
2022-12-01T10:04:14.3222809Z 
2022-12-01T10:04:14.3223044Z COMMIT_MESSAGES=$(git cherry -v "origin/${GIT_DEFAULT_BRANCH:-master}")
2022-12-01T10:04:14.3223265Z 
2022-12-01T10:04:14.3223480Z # sanitize the input commit message and PR body here:
2022-12-01T10:04:14.3223696Z #
2022-12-01T10:04:14.3223970Z # trim all new lines from commit messages + PR_BODY to avoid issues with batch environment
2022-12-01T10:04:14.3224348Z # variable copying. see https://github.com/pytorch/pytorch/pull/80043#issuecomment-1167796028
2022-12-01T10:04:14.3224661Z COMMIT_MESSAGES="${COMMIT_MESSAGES//[$'\n\r']}"
2022-12-01T10:04:14.3224891Z PR_BODY="${PR_BODY//[$'\n\r']}"
2022-12-01T10:04:14.3225064Z 
2022-12-01T10:04:14.3225326Z # then trim all special characters like single and double quotes to avoid unescaped inputs to
2022-12-01T10:04:14.3225602Z # wreak havoc internally
2022-12-01T10:04:14.3225823Z export COMMIT_MESSAGES="${COMMIT_MESSAGES//[\'\"]}"
2022-12-01T10:04:14.3226065Z export PR_BODY="${PR_BODY//[\'\"]}"
2022-12-01T10:04:14.3226255Z 
2022-12-01T10:04:14.3226480Z # detached container should get cleaned up by teardown_ec2_linux
2022-12-01T10:04:14.3226761Z # TODO: Stop building test binaries as part of the build phase
2022-12-01T10:04:14.3227035Z # Used for GPU_FLAG since that doesn't play nice
2022-12-01T10:04:14.3227276Z # shellcheck disable=SC2086,SC2090
2022-12-01T10:04:14.3227485Z container_name=$(docker run \
2022-12-01T10:04:14.3227687Z   ${GPU_FLAG:-} \
2022-12-01T10:04:14.3227888Z   -e BUILD_ENVIRONMENT \
2022-12-01T10:04:14.3228076Z   -e PR_NUMBER \
2022-12-01T10:04:14.3228269Z   -e GITHUB_ACTIONS \
2022-12-01T10:04:14.3228457Z   -e BASE_SHA \
2022-12-01T10:04:14.3228637Z   -e BRANCH \
2022-12-01T10:04:14.3228799Z   -e SHA1 \
2022-12-01T10:04:14.3228990Z   -e AWS_DEFAULT_REGION \
2022-12-01T10:04:14.3229186Z   -e IN_WHEEL_TEST \
2022-12-01T10:04:14.3229362Z   -e SHARD_NUMBER \
2022-12-01T10:04:14.3229552Z   -e TEST_CONFIG \
2022-12-01T10:04:14.3229743Z   -e NUM_TEST_SHARDS \
2022-12-01T10:04:14.3229917Z   -e PR_BODY \
2022-12-01T10:04:14.3230107Z   -e COMMIT_MESSAGES \
2022-12-01T10:04:14.3230321Z   -e PYTORCH_RETRY_TEST_CASES \
2022-12-01T10:04:14.3230535Z   -e PYTORCH_OVERRIDE_FLAKY_SIGNAL \
2022-12-01T10:04:14.3230742Z   -e PR_LABELS \
2022-12-01T10:04:14.3230952Z   -e MAX_JOBS="$(nproc --ignore=2)" \
2022-12-01T10:04:14.3231149Z   -e SCCACHE_BUCKET \
2022-12-01T10:04:14.3231352Z   -e SCCACHE_S3_KEY_PREFIX \
2022-12-01T10:04:14.3231550Z   -e XLA_CUDA \
2022-12-01T10:04:14.3231760Z   -e XLA_CLANG_CACHE_S3_BUCKET_NAME \
2022-12-01T10:04:14.3231998Z   --env-file="/tmp/github_env_${GITHUB_RUN_ID}" \
2022-12-01T10:04:14.3232232Z   --ulimit stack=10485760:83886080 \
2022-12-01T10:04:14.3232463Z   --security-opt seccomp=unconfined \
2022-12-01T10:04:14.3232673Z   --cap-add=SYS_PTRACE \
2022-12-01T10:04:14.3232864Z   --ipc=host \
2022-12-01T10:04:14.3233136Z   --shm-size="${SHM_SIZE}" \
2022-12-01T10:04:14.3233316Z   --tty \
2022-12-01T10:04:14.3233537Z   --detach \
2022-12-01T10:04:14.3233735Z   --name="${container_name}" \
2022-12-01T10:04:14.3233925Z   --user jenkins \
2022-12-01T10:04:14.3234162Z   -v "${GITHUB_WORKSPACE}:/var/lib/jenkins/workspace" \
2022-12-01T10:04:14.3234416Z   -w /var/lib/jenkins/workspace \
2022-12-01T10:04:14.3234619Z   "${DOCKER_IMAGE}"
2022-12-01T10:04:14.3234780Z )
2022-12-01T10:04:14.3235056Z docker exec -t "${container_name}" sh -c "pip install $(echo dist/*.whl)[opt-einsum] && ${TEST_COMMAND}"
2022-12-01T10:04:14.3246444Z shell: /usr/bin/bash -e {0}
2022-12-01T10:04:14.3246632Z env:
2022-12-01T10:04:14.3246808Z   GIT_DEFAULT_BRANCH: master
2022-12-01T10:04:14.3247029Z   BUILD_ENVIRONMENT: linux-focal-py3.7-gcc7
2022-12-01T10:04:14.3247243Z   PR_NUMBER: 89997
2022-12-01T10:04:14.3247427Z   BRANCH: pull/89997
2022-12-01T10:04:14.3247634Z   SHA1: c13d400bffe90e16b96520bbc8a41a6f0c9cd584
2022-12-01T10:04:14.3247880Z   BASE_SHA: ae2fe4033cf3b17259b17f351020b988fa893f91
2022-12-01T10:04:14.3248106Z   PYTORCH_RETRY_TEST_CASES: 1
2022-12-01T10:04:14.3248306Z   PYTORCH_OVERRIDE_FLAKY_SIGNAL: 1
2022-12-01T10:04:14.3248513Z   TEST_CONFIG: distributed
2022-12-01T10:04:14.3248704Z   SHARD_NUMBER: 2
2022-12-01T10:04:14.3248879Z   NUM_TEST_SHARDS: 2
2022-12-01T10:04:14.3249534Z   PR_BODY: Link to landed master PR (if applicable):
https://github.com/pytorch/pytorch/pull/88993

Criteria category:
1: This prevents a crash, which was introduced [here](https://github.com/pytorch/pytorch/commit/4b7de265569f7fd731dd1cfea83ce804cc22f7c0#diff-45b117ca26b4fc9174fbe0d7a9cd8cb1c43964cd5e2bb20c7778ee00a942ef63), tagged for 1.13
2: Prevents a crash

I'm hoping this is a low-risk change, since it's just changing one method for its safer form.
2022-12-01T10:04:14.3250225Z   SCCACHE_BUCKET: ossci-compiler-cache-circleci-v2
2022-12-01T10:04:14.3250471Z   SCCACHE_S3_KEY_PREFIX: pull
2022-12-01T10:04:14.3250658Z   SHM_SIZE: 1g
2022-12-01T10:04:14.3250983Z   DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-py3.7-gcc7:fa72f5a0a230eb632055220542038bd4ceca184b
2022-12-01T10:04:14.3251304Z   XLA_CUDA: 
2022-12-01T10:04:14.3251565Z   XLA_CLANG_CACHE_S3_BUCKET_NAME: ossci-compiler-clang-cache-circleci-xla
2022-12-01T10:04:14.3251859Z ##[endgroup]
2022-12-01T10:04:14.3276412Z + [[ distributed == \m\u\l\t\i\g\p\u ]]
2022-12-01T10:04:14.3276830Z + [[ linux-focal-py3.7-gcc7 == *onnx* ]]
2022-12-01T10:04:14.3277067Z + TEST_COMMAND=.jenkins/pytorch/test.sh
2022-12-01T10:04:14.3279517Z ++ git cherry -v origin/master
2022-12-01T10:04:14.7171170Z + COMMIT_MESSAGES='+ 56508e29e6b1847e050462d818021a0389d0de99 Release 1.13, Install torch from test channel, Pin build… (#86290)
2022-12-01T10:04:14.7171928Z - f682048bf9475d324044e955d8d67464ee446c3a Fix for the binary upload (#86385)
2022-12-01T10:04:14.7172734Z + 95112ca043c37e38101e629c1f69663e911cc909 Fix binary builds for the release - unblock release (#86484)
2022-12-01T10:04:14.7173282Z - c38dbd0e1dd88cce783c8dce6cce1a97276b6bb9 Conditionally build the TestApp benchmark based on lite interpreter (#86314) (#86377)
2022-12-01T10:04:14.7173763Z - 50a9cd95baf29e136fdf5f027c6f4c4a7ccd5b9b Add version selector back to functorch docs (#86602) (#86689)
2022-12-01T10:04:14.7174227Z - 5be45fc4f1a86624fe5ce0e99a2a5558011ca69e [ROCm] set nvfuser default to disabled, keep CI (#86369) (#86725)
2022-12-01T10:04:14.7174676Z - f37023b03f3e92dadb247e89fd4e024eb4a0eb8a [MPS] Better error message for `slow_conv2d_forward` (#86844)
2022-12-01T10:04:14.7175120Z - 786431cd13d171965feeb17240cc7bf8dc509dba [DOC] Use type hints to show annotation in the docs  (#86851)
2022-12-01T10:04:14.7175570Z - 03992a6fb3dbd371fe41d3cd95c6589b45a10f14 Make the data types of output and input consistenst for batchnorm (#86784)
2022-12-01T10:04:14.7175927Z + d9ddab5efa2224704e4b982e651589b31b1c1547 [1.13] Remove torch.vmap (#86333)
2022-12-01T10:04:14.7176536Z + 01042487e2d848d17351555ea73c61d38a60a7f9 [1.13] Release-only improvements to functorch docs (#86693)
2022-12-01T10:04:14.7177025Z - 71251e2521ac4b737af43d536dff359df256eadd ci: Just use regular checkout (#86824) (#86895)
2022-12-01T10:04:14.7177357Z + f65ac2ad20f8eab95a0d0fb93c3550afc2b86d91 [CI] Fix builder ref for release, linux only (#86904)
2022-12-01T10:04:14.7177830Z - 51c16f092ca8d7802decfd56d34ae99d1c0f335c [functorch] Add more details to the functorch install page (#86823) (#86903)
2022-12-01T10:04:14.7178326Z - 311b47b72c73509a82ed0b613a7c563e6628332b [quant] Move the order of x86 engine to avoid changing the default qengine (#86631) (#86726)
2022-12-01T10:04:14.7178723Z + 7e196948ca0c5ede7fa6e0831fda18135e6573a5 Add vmap support for slogdet; fix regression from functorch 0.2.1 (#86815) (#86902)
2022-12-01T10:04:14.7179200Z - aab841e89d850c9da9048a64541cfa78b222f1ff [ONNX] Support device().type() string comparison with constant (#86168) (#86921)
2022-12-01T10:04:14.7179743Z - de3fa48e38e5370d18748e131bd20db53390ea5c Install c10d headers with absolute path (#86257) (#86933)
2022-12-01T10:04:14.7180102Z + 366c59e2606368a6f2ee7165d58686d0d494dd92 Allow PrivateUse1 backends to not have Storage (#86557) (#86803)
2022-12-01T10:04:14.7180565Z + 492d572086a93a31fc8972d29726453ee1e46717 [functorch] fix cross (#86934)
2022-12-01T10:04:14.7181058Z - f89a762a8efcca77722071ec042a02742ec4537d Fix the performance issue that the for-loop before ExternallCall could not be parallelized. (#85056) (#86516)
2022-12-01T10:04:14.7181571Z - c40a04415c59edc45b2ceba682377b255f2091c7 Add error checking to flaky test bot platform parser (#86632) (#87201)
2022-12-01T10:04:14.7181953Z + eda8263f151a17d0d81d33e3f48fb2102daf4c14 [Release 1.13.0][DataPipe] Documentation and interface fix (#87140)
2022-12-01T10:04:14.7182297Z + d2325f3b6a17e54b5d48c68d82b74354dfd35f61 handle libomp update on circleci (#86979) (#87132)
2022-12-01T10:04:14.7182735Z - 7688e5eeabce25b67aa99280772d094b1c5f7091 [ONNX] Fix triu/tril export with diagonal input (#86843) (#86925)
2022-12-01T10:04:14.7183185Z - 1ebff1dcaf75d0305dc713eea104ddfa3ae6b588 [ONNX] Renable assert diagnostic test (#85999) (#86924)
2022-12-01T10:04:14.7183643Z - 1442f04f3bab3b0e0f22fbb68c8f328e17801c69 Enables lazy loading of cuda modules if not set by user (#86509)
2022-12-01T10:04:14.7183984Z + d324341922beb064057e12daae19199480885d5e Bug fix for torch._C._willEngineExecuteNode (#86672)
2022-12-01T10:04:14.7184457Z - 894bad72c9dfb946e9bdfebaf2db41473b3dd4fe [ONNX] Fix scalar_type_analysis metadata for copied constant (#86716) (#86923)
2022-12-01T10:04:14.7184924Z - ef6dd0d34d379604eec8db34dd58581b8b061249 [ONNX] Ignore print(Tensor) during tracing (#86223) (#87122)
2022-12-01T10:04:14.7185378Z - e6d8c1392c817636c15ad974dd160292cdc471bc [einsum] Fix opt_einsum defaults to be more reasonable (#86985) (#87144)
2022-12-01T10:04:14.7185758Z + 3af7c7fc98af441ca90ef38b262497b530a317fb [einsum] fix MPS regression and fix incorrect contraction order when path is None (#87261)
2022-12-01T10:04:14.7186228Z - 9e3df4920f75fa2aaf5bbe04ed302fc80008a8b9 Improve NestedTensor documentation (#85186) (#87337)
2022-12-01T10:04:14.7186684Z - 3ca59f4b8aff32fbad3fcd13b812b5335047b8ce Reenable aot tests on windows for cuda 11.7 and up (#87193) (#87307)
2022-12-01T10:04:14.7187114Z - 0948dbb3e7d6e5e99305b71423f7e9a45a6e5e14 Add torch.sparse overview section (#85265) (#87376)
2022-12-01T10:04:14.7187581Z - 1f80ac7fa72cacafbb4901109b465db05f91d818 [Docs] Update mm family ops and F.linear to note limited sparse support.  (#86220) (#87379)
2022-12-01T10:04:14.7188070Z - 7342a3b0eac2e3ebf4d9b5ce1c3c73c4134c8751 Add prototype warning to MaskedTensor constructor (#87107) (#87380)
2022-12-01T10:04:14.7188502Z - aba5544affda233f8413b5fd74df1cdf7ed91c18 [maskedtensor] add docs (#84887) (#87381)
2022-12-01T10:04:14.7188965Z - 6e2cb22d74d1939e0aa5c00c665a58e8796bef31 [functorch][docs] Downgrade the warning about forward-mode AD coverage (#87383) (#87386)
2022-12-01T10:04:14.7189452Z + ca0ee922bc9440bfc86b332d69ee0559f748bc6e [release] Add warning to stateless.functional_call for deprecated behavior (#87079)
2022-12-01T10:04:14.7189978Z - d2ce4310bad94f2e30fd75cb14e175bcef7ba6fa Assert if padding mask type is unexpected (#87106) (#87384)
2022-12-01T10:04:14.7190432Z - 05cb998e11d109a55ddee0fb72c17aa07fa8104a [MPS] Do not dispatch empty job in  `bitwise_not` (#87286) (#87427)
2022-12-01T10:04:14.7190838Z - 385022c76e4229a9393d63871225e0961e91c2db [ci] handle libomp upgrade on github (#87382) (#87408)
2022-12-01T10:04:14.7191254Z - 3ffddc0b2d5fc963463c98b518b0eb1f88788a74 [functorch] fix AOTAutograd tutorial (#87415) (#87434)
2022-12-01T10:04:14.7191590Z + 55c76baf579cb6593f87d1a23e9a49afeb55f15a [1.13] Fix functorch version docs (#87392)
2022-12-01T10:04:14.7191986Z - 59686b4c60289188436ac61576fd389821858fa5 [maskedtensor] fix docs formatting (#87387) (#87406)
2022-12-01T10:04:14.7192307Z + 0c0df0be7497112022804df03aeeb0fcbadc9243 Add `weights_only` option to `torch.load` (#87443)
2022-12-01T10:04:14.7192657Z + d253eb29d86af51ae17b950825ffdb5661b5af7f Avoid calling logging.basicConfig (#86959) (#87455)
2022-12-01T10:04:14.7203303Z - d3aecbd9bc58a42366abffa63748d73c872bd927 Delete torch::deploy from pytorch core (#85953) (#85953) (#87454)
2022-12-01T10:04:14.7203783Z - 51fa4fae41367a17f49ce648c7cdd6aa72f6e6ac Move PadNd from ATen/native to ATen (#87456)
2022-12-01T10:04:14.7204248Z - f6c42ae2c29ba788523149dbcc791bf14530f93d Reenable `isinstance` with `torch.distributed.ReduceOp` (#87303) (#87463)
2022-12-01T10:04:14.7204771Z - 6a8be2cb630ce79d654a707e5ea454d013acbda1 [ONNX] Reland: Update training state logic to support ScriptedModule (#86745) (#87457)
2022-12-01T10:04:14.7205154Z + 8569a44f38ff906103f40c6925dec366c7557943 [MPS] Revamp copy_to_mps_ implementation (#87475)
2022-12-01T10:04:14.7205599Z - fdb18da4cc3ee04c2423eb33b2780edc4cae64e0 Fix distributed issue by including distributed files (#87612)
2022-12-01T10:04:14.7206020Z - 341c377c0f595ff4e4a0fbdef21978c038b64b98 Add General Project Policies (#87385) (#87613)
2022-12-01T10:04:14.7206407Z - 4e1a4b150a4ccd5f401a4293a0336a9a8a1dad2d fix docs push (#87498) (#87628)
2022-12-01T10:04:14.7206829Z - 7c98e70d44abc7a1aead68b6ea6c8adc8c554db5 attempted fix for nvrtc with lovelace (#87611) (#87618)
2022-12-01T10:04:14.7207267Z - 74a9ca993bd79f8131829e9c946657fa9a1d05ef [JIT][Security] Do not blindly eval input string (#89189) (#89925)
2022-12-01T10:04:14.7207594Z + ae2fe4033cf3b17259b17f351020b988fa893f91 Update masked.rst (#89758) (#89923)
2022-12-01T10:04:14.7208001Z - c13d400bffe90e16b96520bbc8a41a6f0c9cd584 Use the Python frame safely in _pythonCallstack'
2022-12-01T10:04:14.7230930Z + COMMIT_MESSAGES='+ 56508e29e6b1847e050462d818021a0389d0de99 Release 1.13, Install torch from test channel, Pin build… (#86290)- f682048bf9475d324044e955d8d67464ee446c3a Fix for the binary upload (#86385)+ 95112ca043c37e38101e629c1f69663e911cc909 Fix binary builds for the release - unblock release (#86484)- c38dbd0e1dd88cce783c8dce6cce1a97276b6bb9 Conditionally build the TestApp benchmark based on lite interpreter (#86314) (#86377)- 50a9cd95baf29e136fdf5f027c6f4c4a7ccd5b9b Add version selector back to functorch docs (#86602) (#86689)- 5be45fc4f1a86624fe5ce0e99a2a5558011ca69e [ROCm] set nvfuser default to disabled, keep CI (#86369) (#86725)- f37023b03f3e92dadb247e89fd4e024eb4a0eb8a [MPS] Better error message for `slow_conv2d_forward` (#86844)- 786431cd13d171965feeb17240cc7bf8dc509dba [DOC] Use type hints to show annotation in the docs  (#86851)- 03992a6fb3dbd371fe41d3cd95c6589b45a10f14 Make the data types of output and input consistenst for batchnorm (#86784)+ d9ddab5efa2224704e4b982e651589b31b1c1547 [1.13] Remove torch.vmap (#86333)+ 01042487e2d848d17351555ea73c61d38a60a7f9 [1.13] Release-only improvements to functorch docs (#86693)- 71251e2521ac4b737af43d536dff359df256eadd ci: Just use regular checkout (#86824) (#86895)+ f65ac2ad20f8eab95a0d0fb93c3550afc2b86d91 [CI] Fix builder ref for release, linux only (#86904)- 51c16f092ca8d7802decfd56d34ae99d1c0f335c [functorch] Add more details to the functorch install page (#86823) (#86903)- 311b47b72c73509a82ed0b613a7c563e6628332b [quant] Move the order of x86 engine to avoid changing the default qengine (#86631) (#86726)+ 7e196948ca0c5ede7fa6e0831fda18135e6573a5 Add vmap support for slogdet; fix regression from functorch 0.2.1 (#86815) (#86902)- aab841e89d850c9da9048a64541cfa78b222f1ff [ONNX] Support device().type() string comparison with constant (#86168) (#86921)- de3fa48e38e5370d18748e131bd20db53390ea5c Install c10d headers with absolute path (#86257) (#86933)+ 366c59e2606368a6f2ee7165d58686d0d494dd92 Allow PrivateUse1 backends to not have Storage (#86557) (#86803)+ 492d572086a93a31fc8972d29726453ee1e46717 [functorch] fix cross (#86934)- f89a762a8efcca77722071ec042a02742ec4537d Fix the performance issue that the for-loop before ExternallCall could not be parallelized. (#85056) (#86516)- c40a04415c59edc45b2ceba682377b255f2091c7 Add error checking to flaky test bot platform parser (#86632) (#87201)+ eda8263f151a17d0d81d33e3f48fb2102daf4c14 [Release 1.13.0][DataPipe] Documentation and interface fix (#87140)+ d2325f3b6a17e54b5d48c68d82b74354dfd35f61 handle libomp update on circleci (#86979) (#87132)- 7688e5eeabce25b67aa99280772d094b1c5f7091 [ONNX] Fix triu/tril export with diagonal input (#86843) (#86925)- 1ebff1dcaf75d0305dc713eea104ddfa3ae6b588 [ONNX] Renable assert diagnostic test (#85999) (#86924)- 1442f04f3bab3b0e0f22fbb68c8f328e17801c69 Enables lazy loading of cuda modules if not set by user (#86509)+ d324341922beb064057e12daae19199480885d5e Bug fix for torch._C._willEngineExecuteNode (#86672)- 894bad72c9dfb946e9bdfebaf2db41473b3dd4fe [ONNX] Fix scalar_type_analysis metadata for copied constant (#86716) (#86923)- ef6dd0d34d379604eec8db34dd58581b8b061249 [ONNX] Ignore print(Tensor) during tracing (#86223) (#87122)- e6d8c1392c817636c15ad974dd160292cdc471bc [einsum] Fix opt_einsum defaults to be more reasonable (#86985) (#87144)+ 3af7c7fc98af441ca90ef38b262497b530a317fb [einsum] fix MPS regression and fix incorrect contraction order when path is None (#87261)- 9e3df4920f75fa2aaf5bbe04ed302fc80008a8b9 Improve NestedTensor documentation (#85186) (#87337)- 3ca59f4b8aff32fbad3fcd13b812b5335047b8ce Reenable aot tests on windows for cuda 11.7 and up (#87193) (#87307)- 0948dbb3e7d6e5e99305b71423f7e9a45a6e5e14 Add torch.sparse overview section (#85265) (#87376)- 1f80ac7fa72cacafbb4901109b465db05f91d818 [Docs] Update mm family ops and F.linear to note limited sparse support.  (#86220) (#87379)- 7342a3b0eac2e3ebf4d9b5ce1c3c73c4134c8751 Add prototype warning to MaskedTensor constructor (#87107) (#87380)- aba5544affda233f8413b5fd74df1cdf7ed91c18 [maskedtensor] add docs (#84887) (#87381)- 6e2cb22d74d1939e0aa5c00c665a58e8796bef31 [functorch][docs] Downgrade the warning about forward-mode AD coverage (#87383) (#87386)+ ca0ee922bc9440bfc86b332d69ee0559f748bc6e [release] Add warning to stateless.functional_call for deprecated behavior (#87079)- d2ce4310bad94f2e30fd75cb14e175bcef7ba6fa Assert if padding mask type is unexpected (#87106) (#87384)- 05cb998e11d109a55ddee0fb72c17aa07fa8104a [MPS] Do not dispatch empty job in  `bitwise_not` (#87286) (#87427)- 385022c76e4229a9393d63871225e0961e91c2db [ci] handle libomp upgrade on github (#87382) (#87408)- 3ffddc0b2d5fc963463c98b518b0eb1f88788a74 [functorch] fix AOTAutograd tutorial (#87415) (#87434)+ 55c76baf579cb6593f87d1a23e9a49afeb55f15a [1.13] Fix functorch version docs (#87392)- 59686b4c60289188436ac61576fd389821858fa5 [maskedtensor] fix docs formatting (#87387) (#87406)+ 0c0df0be7497112022804df03aeeb0fcbadc9243 Add `weights_only` option to `torch.load` (#87443)+ d253eb29d86af51ae17b950825ffdb5661b5af7f Avoid calling logging.basicConfig (#86959) (#87455)- d3aecbd9bc58a42366abffa63748d73c872bd927 Delete torch::deploy from pytorch core (#85953) (#85953) (#87454)- 51fa4fae41367a17f49ce648c7cdd6aa72f6e6ac Move PadNd from ATen/native to ATen (#87456)- f6c42ae2c29ba788523149dbcc791bf14530f93d Reenable `isinstance` with `torch.distributed.ReduceOp` (#87303) (#87463)- 6a8be2cb630ce79d654a707e5ea454d013acbda1 [ONNX] Reland: Update training state logic to support ScriptedModule (#86745) (#87457)+ 8569a44f38ff906103f40c6925dec366c7557943 [MPS] Revamp copy_to_mps_ implementation (#87475)- fdb18da4cc3ee04c2423eb33b2780edc4cae64e0 Fix distributed issue by including distributed files (#87612)- 341c377c0f595ff4e4a0fbdef21978c038b64b98 Add General Project Policies (#87385) (#87613)- 4e1a4b150a4ccd5f401a4293a0336a9a8a1dad2d fix docs push (#87498) (#87628)- 7c98e70d44abc7a1aead68b6ea6c8adc8c554db5 attempted fix for nvrtc with lovelace (#87611) (#87618)- 74a9ca993bd79f8131829e9c946657fa9a1d05ef [JIT][Security] Do not blindly eval input string (#89189) (#89925)+ ae2fe4033cf3b17259b17f351020b988fa893f91 Update masked.rst (#89758) (#89923)- c13d400bffe90e16b96520bbc8a41a6f0c9cd584 Use the Python frame safely in _pythonCallstack'
2022-12-01T10:04:14.7237813Z + PR_BODY='Link to landed master PR (if applicable):https://github.com/pytorch/pytorch/pull/88993Criteria category:1: This prevents a crash, which was introduced [here](https://github.com/pytorch/pytorch/commit/4b7de265569f7fd731dd1cfea83ce804cc22f7c0#diff-45b117ca26b4fc9174fbe0d7a9cd8cb1c43964cd5e2bb20c7778ee00a942ef63), tagged for 1.132: Prevents a crashI'\''m hoping this is a low-risk change, since it'\''s just changing one method for its safer form.'
2022-12-01T10:04:14.7251144Z + export 'COMMIT_MESSAGES=+ 56508e29e6b1847e050462d818021a0389d0de99 Release 1.13, Install torch from test channel, Pin build… (#86290)- f682048bf9475d324044e955d8d67464ee446c3a Fix for the binary upload (#86385)+ 95112ca043c37e38101e629c1f69663e911cc909 Fix binary builds for the release - unblock release (#86484)- c38dbd0e1dd88cce783c8dce6cce1a97276b6bb9 Conditionally build the TestApp benchmark based on lite interpreter (#86314) (#86377)- 50a9cd95baf29e136fdf5f027c6f4c4a7ccd5b9b Add version selector back to functorch docs (#86602) (#86689)- 5be45fc4f1a86624fe5ce0e99a2a5558011ca69e [ROCm] set nvfuser default to disabled, keep CI (#86369) (#86725)- f37023b03f3e92dadb247e89fd4e024eb4a0eb8a [MPS] Better error message for `slow_conv2d_forward` (#86844)- 786431cd13d171965feeb17240cc7bf8dc509dba [DOC] Use type hints to show annotation in the docs  (#86851)- 03992a6fb3dbd371fe41d3cd95c6589b45a10f14 Make the data types of output and input consistenst for batchnorm (#86784)+ d9ddab5efa2224704e4b982e651589b31b1c1547 [1.13] Remove torch.vmap (#86333)+ 01042487e2d848d17351555ea73c61d38a60a7f9 [1.13] Release-only improvements to functorch docs (#86693)- 71251e2521ac4b737af43d536dff359df256eadd ci: Just use regular checkout (#86824) (#86895)+ f65ac2ad20f8eab95a0d0fb93c3550afc2b86d91 [CI] Fix builder ref for release, linux only (#86904)- 51c16f092ca8d7802decfd56d34ae99d1c0f335c [functorch] Add more details to the functorch install page (#86823) (#86903)- 311b47b72c73509a82ed0b613a7c563e6628332b [quant] Move the order of x86 engine to avoid changing the default qengine (#86631) (#86726)+ 7e196948ca0c5ede7fa6e0831fda18135e6573a5 Add vmap support for slogdet; fix regression from functorch 0.2.1 (#86815) (#86902)- aab841e89d850c9da9048a64541cfa78b222f1ff [ONNX] Support device().type() string comparison with constant (#86168) (#86921)- de3fa48e38e5370d18748e131bd20db53390ea5c Install c10d headers with absolute path (#86257) (#86933)+ 366c59e2606368a6f2ee7165d58686d0d494dd92 Allow PrivateUse1 backends to not have Storage (#86557) (#86803)+ 492d572086a93a31fc8972d29726453ee1e46717 [functorch] fix cross (#86934)- f89a762a8efcca77722071ec042a02742ec4537d Fix the performance issue that the for-loop before ExternallCall could not be parallelized. (#85056) (#86516)- c40a04415c59edc45b2ceba682377b255f2091c7 Add error checking to flaky test bot platform parser (#86632) (#87201)+ eda8263f151a17d0d81d33e3f48fb2102daf4c14 [Release 1.13.0][DataPipe] Documentation and interface fix (#87140)+ d2325f3b6a17e54b5d48c68d82b74354dfd35f61 handle libomp update on circleci (#86979) (#87132)- 7688e5eeabce25b67aa99280772d094b1c5f7091 [ONNX] Fix triu/tril export with diagonal input (#86843) (#86925)- 1ebff1dcaf75d0305dc713eea104ddfa3ae6b588 [ONNX] Renable assert diagnostic test (#85999) (#86924)- 1442f04f3bab3b0e0f22fbb68c8f328e17801c69 Enables lazy loading of cuda modules if not set by user (#86509)+ d324341922beb064057e12daae19199480885d5e Bug fix for torch._C._willEngineExecuteNode (#86672)- 894bad72c9dfb946e9bdfebaf2db41473b3dd4fe [ONNX] Fix scalar_type_analysis metadata for copied constant (#86716) (#86923)- ef6dd0d34d379604eec8db34dd58581b8b061249 [ONNX] Ignore print(Tensor) during tracing (#86223) (#87122)- e6d8c1392c817636c15ad974dd160292cdc471bc [einsum] Fix opt_einsum defaults to be more reasonable (#86985) (#87144)+ 3af7c7fc98af441ca90ef38b262497b530a317fb [einsum] fix MPS regression and fix incorrect contraction order when path is None (#87261)- 9e3df4920f75fa2aaf5bbe04ed302fc80008a8b9 Improve NestedTensor documentation (#85186) (#87337)- 3ca59f4b8aff32fbad3fcd13b812b5335047b8ce Reenable aot tests on windows for cuda 11.7 and up (#87193) (#87307)- 0948dbb3e7d6e5e99305b71423f7e9a45a6e5e14 Add torch.sparse overview section (#85265) (#87376)- 1f80ac7fa72cacafbb4901109b465db05f91d818 [Docs] Update mm family ops and F.linear to note limited sparse support.  (#86220) (#87379)- 7342a3b0eac2e3ebf4d9b5ce1c3c73c4134c8751 Add prototype warning to MaskedTensor constructor (#87107) (#87380)- aba5544affda233f8413b5fd74df1cdf7ed91c18 [maskedtensor] add docs (#84887) (#87381)- 6e2cb22d74d1939e0aa5c00c665a58e8796bef31 [functorch][docs] Downgrade the warning about forward-mode AD coverage (#87383) (#87386)+ ca0ee922bc9440bfc86b332d69ee0559f748bc6e [release] Add warning to stateless.functional_call for deprecated behavior (#87079)- d2ce4310bad94f2e30fd75cb14e175bcef7ba6fa Assert if padding mask type is unexpected (#87106) (#87384)- 05cb998e11d109a55ddee0fb72c17aa07fa8104a [MPS] Do not dispatch empty job in  `bitwise_not` (#87286) (#87427)- 385022c76e4229a9393d63871225e0961e91c2db [ci] handle libomp upgrade on github (#87382) (#87408)- 3ffddc0b2d5fc963463c98b518b0eb1f88788a74 [functorch] fix AOTAutograd tutorial (#87415) (#87434)+ 55c76baf579cb6593f87d1a23e9a49afeb55f15a [1.13] Fix functorch version docs (#87392)- 59686b4c60289188436ac61576fd389821858fa5 [maskedtensor] fix docs formatting (#87387) (#87406)+ 0c0df0be7497112022804df03aeeb0fcbadc9243 Add `weights_only` option to `torch.load` (#87443)+ d253eb29d86af51ae17b950825ffdb5661b5af7f Avoid calling logging.basicConfig (#86959) (#87455)- d3aecbd9bc58a42366abffa63748d73c872bd927 Delete torch::deploy from pytorch core (#85953) (#85953) (#87454)- 51fa4fae41367a17f49ce648c7cdd6aa72f6e6ac Move PadNd from ATen/native to ATen (#87456)- f6c42ae2c29ba788523149dbcc791bf14530f93d Reenable `isinstance` with `torch.distributed.ReduceOp` (#87303) (#87463)- 6a8be2cb630ce79d654a707e5ea454d013acbda1 [ONNX] Reland: Update training state logic to support ScriptedModule (#86745) (#87457)+ 8569a44f38ff906103f40c6925dec366c7557943 [MPS] Revamp copy_to_mps_ implementation (#87475)- fdb18da4cc3ee04c2423eb33b2780edc4cae64e0 Fix distributed issue by including distributed files (#87612)- 341c377c0f595ff4e4a0fbdef21978c038b64b98 Add General Project Policies (#87385) (#87613)- 4e1a4b150a4ccd5f401a4293a0336a9a8a1dad2d fix docs push (#87498) (#87628)- 7c98e70d44abc7a1aead68b6ea6c8adc8c554db5 attempted fix for nvrtc with lovelace (#87611) (#87618)- 74a9ca993bd79f8131829e9c946657fa9a1d05ef [JIT][Security] Do not blindly eval input string (#89189) (#89925)+ ae2fe4033cf3b17259b17f351020b988fa893f91 Update masked.rst (#89758) (#89923)- c13d400bffe90e16b96520bbc8a41a6f0c9cd584 Use the Python frame safely in _pythonCallstack'
2022-12-01T10:04:14.7271088Z + COMMIT_MESSAGES='+ 56508e29e6b1847e050462d818021a0389d0de99 Release 1.13, Install torch from test channel, Pin build… (#86290)- f682048bf9475d324044e955d8d67464ee446c3a Fix for the binary upload (#86385)+ 95112ca043c37e38101e629c1f69663e911cc909 Fix binary builds for the release - unblock release (#86484)- c38dbd0e1dd88cce783c8dce6cce1a97276b6bb9 Conditionally build the TestApp benchmark based on lite interpreter (#86314) (#86377)- 50a9cd95baf29e136fdf5f027c6f4c4a7ccd5b9b Add version selector back to functorch docs (#86602) (#86689)- 5be45fc4f1a86624fe5ce0e99a2a5558011ca69e [ROCm] set nvfuser default to disabled, keep CI (#86369) (#86725)- f37023b03f3e92dadb247e89fd4e024eb4a0eb8a [MPS] Better error message for `slow_conv2d_forward` (#86844)- 786431cd13d171965feeb17240cc7bf8dc509dba [DOC] Use type hints to show annotation in the docs  (#86851)- 03992a6fb3dbd371fe41d3cd95c6589b45a10f14 Make the data types of output and input consistenst for batchnorm (#86784)+ d9ddab5efa2224704e4b982e651589b31b1c1547 [1.13] Remove torch.vmap (#86333)+ 01042487e2d848d17351555ea73c61d38a60a7f9 [1.13] Release-only improvements to functorch docs (#86693)- 71251e2521ac4b737af43d536dff359df256eadd ci: Just use regular checkout (#86824) (#86895)+ f65ac2ad20f8eab95a0d0fb93c3550afc2b86d91 [CI] Fix builder ref for release, linux only (#86904)- 51c16f092ca8d7802decfd56d34ae99d1c0f335c [functorch] Add more details to the functorch install page (#86823) (#86903)- 311b47b72c73509a82ed0b613a7c563e6628332b [quant] Move the order of x86 engine to avoid changing the default qengine (#86631) (#86726)+ 7e196948ca0c5ede7fa6e0831fda18135e6573a5 Add vmap support for slogdet; fix regression from functorch 0.2.1 (#86815) (#86902)- aab841e89d850c9da9048a64541cfa78b222f1ff [ONNX] Support device().type() string comparison with constant (#86168) (#86921)- de3fa48e38e5370d18748e131bd20db53390ea5c Install c10d headers with absolute path (#86257) (#86933)+ 366c59e2606368a6f2ee7165d58686d0d494dd92 Allow PrivateUse1 backends to not have Storage (#86557) (#86803)+ 492d572086a93a31fc8972d29726453ee1e46717 [functorch] fix cross (#86934)- f89a762a8efcca77722071ec042a02742ec4537d Fix the performance issue that the for-loop before ExternallCall could not be parallelized. (#85056) (#86516)- c40a04415c59edc45b2ceba682377b255f2091c7 Add error checking to flaky test bot platform parser (#86632) (#87201)+ eda8263f151a17d0d81d33e3f48fb2102daf4c14 [Release 1.13.0][DataPipe] Documentation and interface fix (#87140)+ d2325f3b6a17e54b5d48c68d82b74354dfd35f61 handle libomp update on circleci (#86979) (#87132)- 7688e5eeabce25b67aa99280772d094b1c5f7091 [ONNX] Fix triu/tril export with diagonal input (#86843) (#86925)- 1ebff1dcaf75d0305dc713eea104ddfa3ae6b588 [ONNX] Renable assert diagnostic test (#85999) (#86924)- 1442f04f3bab3b0e0f22fbb68c8f328e17801c69 Enables lazy loading of cuda modules if not set by user (#86509)+ d324341922beb064057e12daae19199480885d5e Bug fix for torch._C._willEngineExecuteNode (#86672)- 894bad72c9dfb946e9bdfebaf2db41473b3dd4fe [ONNX] Fix scalar_type_analysis metadata for copied constant (#86716) (#86923)- ef6dd0d34d379604eec8db34dd58581b8b061249 [ONNX] Ignore print(Tensor) during tracing (#86223) (#87122)- e6d8c1392c817636c15ad974dd160292cdc471bc [einsum] Fix opt_einsum defaults to be more reasonable (#86985) (#87144)+ 3af7c7fc98af441ca90ef38b262497b530a317fb [einsum] fix MPS regression and fix incorrect contraction order when path is None (#87261)- 9e3df4920f75fa2aaf5bbe04ed302fc80008a8b9 Improve NestedTensor documentation (#85186) (#87337)- 3ca59f4b8aff32fbad3fcd13b812b5335047b8ce Reenable aot tests on windows for cuda 11.7 and up (#87193) (#87307)- 0948dbb3e7d6e5e99305b71423f7e9a45a6e5e14 Add torch.sparse overview section (#85265) (#87376)- 1f80ac7fa72cacafbb4901109b465db05f91d818 [Docs] Update mm family ops and F.linear to note limited sparse support.  (#86220) (#87379)- 7342a3b0eac2e3ebf4d9b5ce1c3c73c4134c8751 Add prototype warning to MaskedTensor constructor (#87107) (#87380)- aba5544affda233f8413b5fd74df1cdf7ed91c18 [maskedtensor] add docs (#84887) (#87381)- 6e2cb22d74d1939e0aa5c00c665a58e8796bef31 [functorch][docs] Downgrade the warning about forward-mode AD coverage (#87383) (#87386)+ ca0ee922bc9440bfc86b332d69ee0559f748bc6e [release] Add warning to stateless.functional_call for deprecated behavior (#87079)- d2ce4310bad94f2e30fd75cb14e175bcef7ba6fa Assert if padding mask type is unexpected (#87106) (#87384)- 05cb998e11d109a55ddee0fb72c17aa07fa8104a [MPS] Do not dispatch empty job in  `bitwise_not` (#87286) (#87427)- 385022c76e4229a9393d63871225e0961e91c2db [ci] handle libomp upgrade on github (#87382) (#87408)- 3ffddc0b2d5fc963463c98b518b0eb1f88788a74 [functorch] fix AOTAutograd tutorial (#87415) (#87434)+ 55c76baf579cb6593f87d1a23e9a49afeb55f15a [1.13] Fix functorch version docs (#87392)- 59686b4c60289188436ac61576fd389821858fa5 [maskedtensor] fix docs formatting (#87387) (#87406)+ 0c0df0be7497112022804df03aeeb0fcbadc9243 Add `weights_only` option to `torch.load` (#87443)+ d253eb29d86af51ae17b950825ffdb5661b5af7f Avoid calling logging.basicConfig (#86959) (#87455)- d3aecbd9bc58a42366abffa63748d73c872bd927 Delete torch::deploy from pytorch core (#85953) (#85953) (#87454)- 51fa4fae41367a17f49ce648c7cdd6aa72f6e6ac Move PadNd from ATen/native to ATen (#87456)- f6c42ae2c29ba788523149dbcc791bf14530f93d Reenable `isinstance` with `torch.distributed.ReduceOp` (#87303) (#87463)- 6a8be2cb630ce79d654a707e5ea454d013acbda1 [ONNX] Reland: Update training state logic to support ScriptedModule (#86745) (#87457)+ 8569a44f38ff906103f40c6925dec366c7557943 [MPS] Revamp copy_to_mps_ implementation (#87475)- fdb18da4cc3ee04c2423eb33b2780edc4cae64e0 Fix distributed issue by including distributed files (#87612)- 341c377c0f595ff4e4a0fbdef21978c038b64b98 Add General Project Policies (#87385) (#87613)- 4e1a4b150a4ccd5f401a4293a0336a9a8a1dad2d fix docs push (#87498) (#87628)- 7c98e70d44abc7a1aead68b6ea6c8adc8c554db5 attempted fix for nvrtc with lovelace (#87611) (#87618)- 74a9ca993bd79f8131829e9c946657fa9a1d05ef [JIT][Security] Do not blindly eval input string (#89189) (#89925)+ ae2fe4033cf3b17259b17f351020b988fa893f91 Update masked.rst (#89758) (#89923)- c13d400bffe90e16b96520bbc8a41a6f0c9cd584 Use the Python frame safely in _pythonCallstack'
2022-12-01T10:04:14.7277918Z + export 'PR_BODY=Link to landed master PR (if applicable):https://github.com/pytorch/pytorch/pull/88993Criteria category:1: This prevents a crash, which was introduced [here](https://github.com/pytorch/pytorch/commit/4b7de265569f7fd731dd1cfea83ce804cc22f7c0#diff-45b117ca26b4fc9174fbe0d7a9cd8cb1c43964cd5e2bb20c7778ee00a942ef63), tagged for 1.132: Prevents a crashIm hoping this is a low-risk change, since its just changing one method for its safer form.'
2022-12-01T10:04:14.7279327Z + PR_BODY='Link to landed master PR (if applicable):https://github.com/pytorch/pytorch/pull/88993Criteria category:1: This prevents a crash, which was introduced [here](https://github.com/pytorch/pytorch/commit/4b7de265569f7fd731dd1cfea83ce804cc22f7c0#diff-45b117ca26b4fc9174fbe0d7a9cd8cb1c43964cd5e2bb20c7778ee00a942ef63), tagged for 1.132: Prevents a crashIm hoping this is a low-risk change, since its just changing one method for its safer form.'
2022-12-01T10:04:14.7279984Z +++ nproc --ignore=2
2022-12-01T10:04:14.7281498Z ++ docker run -e BUILD_ENVIRONMENT -e PR_NUMBER -e GITHUB_ACTIONS -e BASE_SHA -e BRANCH -e SHA1 -e AWS_DEFAULT_REGION -e IN_WHEEL_TEST -e SHARD_NUMBER -e TEST_CONFIG -e NUM_TEST_SHARDS -e PR_BODY -e COMMIT_MESSAGES -e PYTORCH_RETRY_TEST_CASES -e PYTORCH_OVERRIDE_FLAKY_SIGNAL -e PR_LABELS -e MAX_JOBS=6 -e SCCACHE_BUCKET -e SCCACHE_S3_KEY_PREFIX -e XLA_CUDA -e XLA_CLANG_CACHE_S3_BUCKET_NAME --env-file=/tmp/github_env_3591403534 --ulimit stack=10485760:83886080 --security-opt seccomp=unconfined --cap-add=SYS_PTRACE --ipc=host --shm-size=1g --tty --detach --name= --user jenkins -v /home/ec2-user/actions-runner/_work/pytorch/pytorch:/var/lib/jenkins/workspace -w /var/lib/jenkins/workspace 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-py3.7-gcc7:fa72f5a0a230eb632055220542038bd4ceca184b
2022-12-01T10:04:25.7855007Z + container_name=0bf5643a9780fdbcf8441394ea970b88da0db91cdab4fb259a94a9efce50888d
2022-12-01T10:04:25.7858369Z ++ echo dist/torch-1.13.0a0+gitc13d400-cp37-cp37m-linux_x86_64.whl
2022-12-01T10:04:25.7860261Z + docker exec -t 0bf5643a9780fdbcf8441394ea970b88da0db91cdab4fb259a94a9efce50888d sh -c 'pip install dist/torch-1.13.0a0+gitc13d400-cp37-cp37m-linux_x86_64.whl[opt-einsum] && .jenkins/pytorch/test.sh'
2022-12-01T10:04:26.2194059Z Processing ./dist/torch-1.13.0a0+gitc13d400-cp37-cp37m-linux_x86_64.whl
2022-12-01T10:04:26.2965710Z Requirement already satisfied: typing-extensions in /opt/conda/lib/python3.7/site-packages (from torch==1.13.0a0+gitc13d400) (4.3.0)
2022-12-01T10:04:26.2978136Z Requirement already satisfied: opt-einsum>=3.3 in /opt/conda/lib/python3.7/site-packages (from torch==1.13.0a0+gitc13d400) (3.3.0)
2022-12-01T10:04:26.3049842Z Requirement already satisfied: numpy>=1.7 in /opt/conda/lib/python3.7/site-packages (from opt-einsum>=3.3->torch==1.13.0a0+gitc13d400) (1.18.5)
2022-12-01T10:04:27.0861144Z Installing collected packages: torch
2022-12-01T10:04:33.4124773Z Successfully installed torch-1.13.0a0+gitc13d400
2022-12-01T10:04:33.5624952Z ++ python -c 'import site; print(site.getsitepackages()[0])'
2022-12-01T10:04:33.5788672Z + TORCH_INSTALL_DIR=/opt/conda/lib/python3.7/site-packages/torch
2022-12-01T10:04:33.5789389Z + TORCH_BIN_DIR=/opt/conda/lib/python3.7/site-packages/torch/bin
2022-12-01T10:04:33.5789779Z + TORCH_LIB_DIR=/opt/conda/lib/python3.7/site-packages/torch/lib
2022-12-01T10:04:33.5790202Z + TORCH_TEST_DIR=/opt/conda/lib/python3.7/site-packages/torch/test
2022-12-01T10:04:33.5790439Z + BUILD_DIR=build
2022-12-01T10:04:33.5790630Z + BUILD_RENAMED_DIR=build_renamed
2022-12-01T10:04:33.5790891Z + BUILD_BIN_DIR=build/bin
2022-12-01T10:04:33.5791085Z + export VALGRIND=ON
2022-12-01T10:04:33.5791251Z + VALGRIND=ON
2022-12-01T10:04:33.5791543Z + [[ linux-focal-py3.7-gcc7 == *clang9* ]]
2022-12-01T10:04:33.5791878Z + [[ linux-focal-py3.7-gcc7 != *bazel* ]]
2022-12-01T10:04:33.5792865Z ++ realpath build/custom_test_artifacts
2022-12-01T10:04:33.5799411Z + CUSTOM_TEST_ARTIFACT_BUILD_DIR=/var/lib/jenkins/workspace/build/custom_test_artifacts
2022-12-01T10:04:33.5802162Z ++ dirname .jenkins/pytorch/test.sh
2022-12-01T10:04:33.5807832Z + source .jenkins/pytorch/common.sh
2022-12-01T10:04:33.5810892Z +++ dirname .jenkins/pytorch/common.sh
2022-12-01T10:04:33.5815466Z ++ source .jenkins/pytorch/common_utils.sh
2022-12-01T10:04:33.5817297Z +++ declare -f -t trap_add
2022-12-01T10:04:33.5821480Z ++ set -ex
2022-12-01T10:04:33.5821818Z ++ [[ linux-focal-py3.7-gcc7 == *rocm* ]]
2022-12-01T10:04:33.5822098Z ++ BUILD_TEST_LIBTORCH=0
2022-12-01T10:04:33.5822500Z ++ [[ distributed == *xla* ]]
2022-12-01T10:04:33.5822872Z ++ [[ linux-focal-py3.7-gcc7 == *centos* ]]
2022-12-01T10:04:33.5823191Z ++ [[ linux-focal-py3.7-gcc7 == *linux-bionic* ]]
2022-12-01T10:04:33.5823496Z ++ [[ linux-focal-py3.7-gcc7 == *linux-focal* ]]
2022-12-01T10:04:33.5823759Z ++ which conda
2022-12-01T10:04:33.5830481Z /opt/conda/bin/conda
2022-12-01T10:04:33.5831166Z ++ conda install -q -y cmake
2022-12-01T10:04:36.5115488Z Collecting package metadata (current_repodata.json): ...working... done
2022-12-01T10:04:37.3939156Z Solving environment: ...working... done
2022-12-01T10:04:37.4832581Z 
2022-12-01T10:04:37.4833208Z ## Package Plan ##
2022-12-01T10:04:37.4833471Z 
2022-12-01T10:04:37.4833797Z   environment location: /opt/conda
2022-12-01T10:04:37.4834039Z 
2022-12-01T10:04:37.4834144Z   added / updated specs:
2022-12-01T10:04:37.4834441Z     - cmake
2022-12-01T10:04:37.4834545Z 
2022-12-01T10:04:37.4834565Z 
2022-12-01T10:04:37.4834677Z The following packages will be downloaded:
2022-12-01T10:04:37.4834825Z 
2022-12-01T10:04:37.4834918Z     package                    |            build
2022-12-01T10:04:37.4835179Z     ---------------------------|-----------------
2022-12-01T10:04:37.4835493Z     bzip2-1.0.8                |       h7b6447c_0          78 KB
2022-12-01T10:04:37.4835789Z     c-ares-1.18.1              |       h7f8727e_0         114 KB
2022-12-01T10:04:37.4836091Z     ca-certificates-2022.10.11 |       h06a4308_0         124 KB
2022-12-01T10:04:37.4836403Z     certifi-2022.9.24          |   py37h06a4308_0         154 KB
2022-12-01T10:04:37.4836693Z     cmake-3.22.1               |       h1fce559_0         7.3 MB
2022-12-01T10:04:37.4836966Z     expat-2.4.9                |       h6a678d5_0         156 KB
2022-12-01T10:04:37.4837253Z     krb5-1.19.2                |       hac12032_0         1.2 MB
2022-12-01T10:04:37.4837753Z     libcurl-7.84.0             |       h91b91d3_0         337 KB
2022-12-01T10:04:37.4838049Z     libedit-3.1.20210910       |       h7f8727e_0         166 KB
2022-12-01T10:04:37.4838400Z     libev-4.33                 |       h7f8727e_1         111 KB
2022-12-01T10:04:37.4838682Z     libgcc-ng-11.2.0           |       h1234567_1         5.3 MB
2022-12-01T10:04:37.4838971Z     libgomp-11.2.0             |       h1234567_1         474 KB
2022-12-01T10:04:37.4839250Z     libnghttp2-1.46.0          |       hce63b2e_0         680 KB
2022-12-01T10:04:37.4839541Z     libssh2-1.10.0             |       h8f2d780_0         274 KB
2022-12-01T10:04:37.4839834Z     libstdcxx-ng-11.2.0        |       h1234567_1         4.7 MB
2022-12-01T10:04:37.4840108Z     libuv-1.40.0               |       h7b6447c_0         736 KB
2022-12-01T10:04:37.4840388Z     lz4-c-1.9.3                |       h295c915_1         185 KB
2022-12-01T10:04:37.4840668Z     openssl-1.1.1s             |       h7f8727e_0         3.6 MB
2022-12-01T10:04:37.4840950Z     rhash-1.4.1                |       h3c74f83_1         203 KB
2022-12-01T10:04:37.4841213Z     zstd-1.5.2                 |       ha4553b6_0         488 KB
2022-12-01T10:04:37.4841515Z     ------------------------------------------------------------
2022-12-01T10:04:37.4841757Z                                            Total:        26.3 MB
2022-12-01T10:04:37.4841880Z 
2022-12-01T10:04:37.4841978Z The following NEW packages will be INSTALLED:
2022-12-01T10:04:37.4842123Z 
2022-12-01T10:04:37.4842398Z   bzip2              pkgs/main/linux-64::bzip2-1.0.8-h7b6447c_0 None
2022-12-01T10:04:37.4842766Z   c-ares             pkgs/main/linux-64::c-ares-1.18.1-h7f8727e_0 None
2022-12-01T10:04:37.4843131Z   cmake              pkgs/main/linux-64::cmake-3.22.1-h1fce559_0 None
2022-12-01T10:04:37.4843479Z   expat              pkgs/main/linux-64::expat-2.4.9-h6a678d5_0 None
2022-12-01T10:04:37.4843830Z   krb5               pkgs/main/linux-64::krb5-1.19.2-hac12032_0 None
2022-12-01T10:04:37.4844197Z   libcurl            pkgs/main/linux-64::libcurl-7.84.0-h91b91d3_0 None
2022-12-01T10:04:37.4844580Z   libedit            pkgs/main/linux-64::libedit-3.1.20210910-h7f8727e_0 None
2022-12-01T10:04:37.4844936Z   libev              pkgs/main/linux-64::libev-4.33-h7f8727e_1 None
2022-12-01T10:04:37.4845311Z   libnghttp2         pkgs/main/linux-64::libnghttp2-1.46.0-hce63b2e_0 None
2022-12-01T10:04:37.4845692Z   libssh2            pkgs/main/linux-64::libssh2-1.10.0-h8f2d780_0 None
2022-12-01T10:04:37.4846036Z   libuv              pkgs/main/linux-64::libuv-1.40.0-h7b6447c_0 None
2022-12-01T10:04:37.4846392Z   lz4-c              pkgs/main/linux-64::lz4-c-1.9.3-h295c915_1 None
2022-12-01T10:04:37.4846744Z   rhash              pkgs/main/linux-64::rhash-1.4.1-h3c74f83_1 None
2022-12-01T10:04:37.4847097Z   zstd               pkgs/main/linux-64::zstd-1.5.2-ha4553b6_0 None
2022-12-01T10:04:37.4847245Z 
2022-12-01T10:04:37.4847332Z The following packages will be UPDATED:
2022-12-01T10:04:37.4847467Z 
2022-12-01T10:04:37.4847679Z   ca-certificates                     2022.07.19-h06a4308_0 --> 2022.10.11-h06a4308_0 None
2022-12-01T10:04:37.4848038Z   certifi                          2022.9.14-py37h06a4308_0 --> 2022.9.24-py37h06a4308_0 None
2022-12-01T10:04:37.4848386Z   libgcc-ng                               9.3.0-h5101ec6_17 --> 11.2.0-h1234567_1 None
2022-12-01T10:04:37.4848710Z   libgomp                                 9.3.0-h5101ec6_17 --> 11.2.0-h1234567_1 None
2022-12-01T10:04:37.4849043Z   libstdcxx-ng                            9.3.0-hd4cf53a_17 --> 11.2.0-h1234567_1 None
2022-12-01T10:04:37.4849387Z   openssl                                 1.1.1q-h7f8727e_0 --> 1.1.1s-h7f8727e_0 None
2022-12-01T10:04:37.4849532Z 
2022-12-01T10:04:37.4849536Z 
2022-12-01T10:04:38.6818630Z Preparing transaction: ...working... done
2022-12-01T10:04:39.1051226Z Verifying transaction: ...working... done
2022-12-01T10:04:39.8044705Z Executing transaction: ...working... done
2022-12-01T10:04:39.8984658Z Retrieving notices: ...working... done
2022-12-01T10:04:40.1375579Z ++ [[ linux-focal-py3.7-gcc7 == *centos* ]]
2022-12-01T10:04:40.1375914Z + echo 'Environment variables'
2022-12-01T10:04:40.1376175Z Environment variables
2022-12-01T10:04:40.1376356Z + env
2022-12-01T10:04:40.1384173Z INSTALLED_DB=yes
2022-12-01T10:04:40.1384720Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch
2022-12-01T10:04:40.1385404Z BUILD_ENVIRONMENT=linux-focal-py3.7-gcc7
2022-12-01T10:04:40.1385750Z PYTORCH_OVERRIDE_FLAKY_SIGNAL=1
2022-12-01T10:04:40.1386123Z HOSTNAME=0bf5643a9780
2022-12-01T10:04:40.1386812Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_bc9efd0f-a7ec-45e6-b0aa-46e5f907846d
2022-12-01T10:04:40.1387344Z GITHUB_ACTION=__self
2022-12-01T10:04:40.1387692Z GITHUB_RUN_NUMBER=69319
2022-12-01T10:04:40.1387942Z TEST_CONFIG=distributed
2022-12-01T10:04:40.1388219Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all
2022-12-01T10:04:40.1388541Z GITHUB_TRIGGERING_ACTOR=charlie-wt
2022-12-01T10:04:40.1388744Z GITHUB_REF_TYPE=branch
2022-12-01T10:04:40.1389046Z TORCH_CUDA_ARCH_LIST=Maxwell
2022-12-01T10:04:40.1389447Z BASE_SHA=ae2fe4033cf3b17259b17f351020b988fa893f91
2022-12-01T10:04:40.1389785Z XLA_CUDA=
2022-12-01T10:04:40.1390574Z ***
2022-12-01T10:04:40.1390797Z CARGO_NET_GIT_FETCH_WITH_CLI=true
2022-12-01T10:04:40.1391068Z GITHUB_ACTIONS=true
2022-12-01T10:04:40.1391307Z SHA1=c13d400bffe90e16b96520bbc8a41a6f0c9cd584
2022-12-01T10:04:40.1391671Z GITHUB_SHA=cdc064133ad5e5a46a756ee9218659e8f252e950
2022-12-01T10:04:40.1392028Z UCC_HOME=/usr
2022-12-01T10:04:40.1392355Z GITHUB_REF=refs/pull/89997/merge
2022-12-01T10:04:40.1392684Z SHARD_NUMBER=2
2022-12-01T10:04:40.1392887Z GITHUB_REF_PROTECTED=false
2022-12-01T10:04:40.1393067Z HOME=/var/lib/jenkins
2022-12-01T10:04:40.1393289Z GITHUB_API_URL=https://api.github.com
2022-12-01T10:04:40.1393503Z INSTALLED_THRIFT=
2022-12-01T10:04:40.1393667Z UCX_COMMIT=
2022-12-01T10:04:40.1393848Z SCCACHE_S3_KEY_PREFIX=pull
2022-12-01T10:04:40.1394036Z NUM_TEST_SHARDS=2
2022-12-01T10:04:40.1394194Z UCX_HOME=/usr
2022-12-01T10:04:40.1394620Z GITHUB_STATE=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/save_state_bc9efd0f-a7ec-45e6-b0aa-46e5f907846d
2022-12-01T10:04:40.1394936Z PYTORCH_RETRY_TEST_CASES=1
2022-12-01T10:04:40.1395347Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_bc9efd0f-a7ec-45e6-b0aa-46e5f907846d
2022-12-01T10:04:40.1395801Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json
2022-12-01T10:04:40.1396076Z GITHUB_EVENT_NAME=pull_request
2022-12-01T10:04:40.1396273Z GITHUB_RUN_ID=3591403534
2022-12-01T10:04:40.1396690Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_bc9efd0f-a7ec-45e6-b0aa-46e5f907846d
2022-12-01T10:04:40.1397032Z GITHUB_ACTOR=charlie-wt
2022-12-01T10:04:40.1397221Z PR_NUMBER=89997
2022-12-01T10:04:40.1397382Z DESIRED_CUDA=
2022-12-01T10:04:40.1397561Z GITHUB_RUN_ATTEMPT=1
2022-12-01T10:04:40.1397740Z VALGRIND=ON
2022-12-01T10:04:40.1397954Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql
2022-12-01T10:04:40.1398177Z TERM=xterm
2022-12-01T10:04:40.1398355Z INSTALLED_VISION=yes
2022-12-01T10:04:40.1398525Z BRANCH=pull/89997
2022-12-01T10:04:40.1398714Z OPENSSL_ROOT_DIR=/opt/openssl
2022-12-01T10:04:40.1398914Z CUDA_PATH=/usr/local/cuda
2022-12-01T10:04:40.1399270Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux
2022-12-01T10:04:40.1399573Z GITHUB_SERVER_URL=https://github.com
2022-12-01T10:04:40.1399776Z UCC_COMMIT=
2022-12-01T10:04:40.1399936Z INSTALLED_ANDROID=
2022-12-01T10:04:40.1400106Z SHLVL=1
2022-12-01T10:04:40.1400263Z MAX_JOBS=6
2022-12-01T10:04:40.1412591Z COMMIT_MESSAGES=+ 56508e29e6b1847e050462d818021a0389d0de99 Release 1.13, Install torch from test channel, Pin build… (#86290)- f682048bf9475d324044e955d8d67464ee446c3a Fix for the binary upload (#86385)+ 95112ca043c37e38101e629c1f69663e911cc909 Fix binary builds for the release - unblock release (#86484)- c38dbd0e1dd88cce783c8dce6cce1a97276b6bb9 Conditionally build the TestApp benchmark based on lite interpreter (#86314) (#86377)- 50a9cd95baf29e136fdf5f027c6f4c4a7ccd5b9b Add version selector back to functorch docs (#86602) (#86689)- 5be45fc4f1a86624fe5ce0e99a2a5558011ca69e [ROCm] set nvfuser default to disabled, keep CI (#86369) (#86725)- f37023b03f3e92dadb247e89fd4e024eb4a0eb8a [MPS] Better error message for `slow_conv2d_forward` (#86844)- 786431cd13d171965feeb17240cc7bf8dc509dba [DOC] Use type hints to show annotation in the docs  (#86851)- 03992a6fb3dbd371fe41d3cd95c6589b45a10f14 Make the data types of output and input consistenst for batchnorm (#86784)+ d9ddab5efa2224704e4b982e651589b31b1c1547 [1.13] Remove torch.vmap (#86333)+ 01042487e2d848d17351555ea73c61d38a60a7f9 [1.13] Release-only improvements to functorch docs (#86693)- 71251e2521ac4b737af43d536dff359df256eadd ci: Just use regular checkout (#86824) (#86895)+ f65ac2ad20f8eab95a0d0fb93c3550afc2b86d91 [CI] Fix builder ref for release, linux only (#86904)- 51c16f092ca8d7802decfd56d34ae99d1c0f335c [functorch] Add more details to the functorch install page (#86823) (#86903)- 311b47b72c73509a82ed0b613a7c563e6628332b [quant] Move the order of x86 engine to avoid changing the default qengine (#86631) (#86726)+ 7e196948ca0c5ede7fa6e0831fda18135e6573a5 Add vmap support for slogdet; fix regression from functorch 0.2.1 (#86815) (#86902)- aab841e89d850c9da9048a64541cfa78b222f1ff [ONNX] Support device().type() string comparison with constant (#86168) (#86921)- de3fa48e38e5370d18748e131bd20db53390ea5c Install c10d headers with absolute path (#86257) (#86933)+ 366c59e2606368a6f2ee7165d58686d0d494dd92 Allow PrivateUse1 backends to not have Storage (#86557) (#86803)+ 492d572086a93a31fc8972d29726453ee1e46717 [functorch] fix cross (#86934)- f89a762a8efcca77722071ec042a02742ec4537d Fix the performance issue that the for-loop before ExternallCall could not be parallelized. (#85056) (#86516)- c40a04415c59edc45b2ceba682377b255f2091c7 Add error checking to flaky test bot platform parser (#86632) (#87201)+ eda8263f151a17d0d81d33e3f48fb2102daf4c14 [Release 1.13.0][DataPipe] Documentation and interface fix (#87140)+ d2325f3b6a17e54b5d48c68d82b74354dfd35f61 handle libomp update on circleci (#86979) (#87132)- 7688e5eeabce25b67aa99280772d094b1c5f7091 [ONNX] Fix triu/tril export with diagonal input (#86843) (#86925)- 1ebff1dcaf75d0305dc713eea104ddfa3ae6b588 [ONNX] Renable assert diagnostic test (#85999) (#86924)- 1442f04f3bab3b0e0f22fbb68c8f328e17801c69 Enables lazy loading of cuda modules if not set by user (#86509)+ d324341922beb064057e12daae19199480885d5e Bug fix for torch._C._willEngineExecuteNode (#86672)- 894bad72c9dfb946e9bdfebaf2db41473b3dd4fe [ONNX] Fix scalar_type_analysis metadata for copied constant (#86716) (#86923)- ef6dd0d34d379604eec8db34dd58581b8b061249 [ONNX] Ignore print(Tensor) during tracing (#86223) (#87122)- e6d8c1392c817636c15ad974dd160292cdc471bc [einsum] Fix opt_einsum defaults to be more reasonable (#86985) (#87144)+ 3af7c7fc98af441ca90ef38b262497b530a317fb [einsum] fix MPS regression and fix incorrect contraction order when path is None (#87261)- 9e3df4920f75fa2aaf5bbe04ed302fc80008a8b9 Improve NestedTensor documentation (#85186) (#87337)- 3ca59f4b8aff32fbad3fcd13b812b5335047b8ce Reenable aot tests on windows for cuda 11.7 and up (#87193) (#87307)- 0948dbb3e7d6e5e99305b71423f7e9a45a6e5e14 Add torch.sparse overview section (#85265) (#87376)- 1f80ac7fa72cacafbb4901109b465db05f91d818 [Docs] Update mm family ops and F.linear to note limited sparse support.  (#86220) (#87379)- 7342a3b0eac2e3ebf4d9b5ce1c3c73c4134c8751 Add prototype warning to MaskedTensor constructor (#87107) (#87380)- aba5544affda233f8413b5fd74df1cdf7ed91c18 [maskedtensor] add docs (#84887) (#87381)- 6e2cb22d74d1939e0aa5c00c665a58e8796bef31 [functorch][docs] Downgrade the warning about forward-mode AD coverage (#87383) (#87386)+ ca0ee922bc9440bfc86b332d69ee0559f748bc6e [release] Add warning to stateless.functional_call for deprecated behavior (#87079)- d2ce4310bad94f2e30fd75cb14e175bcef7ba6fa Assert if padding mask type is unexpected (#87106) (#87384)- 05cb998e11d109a55ddee0fb72c17aa07fa8104a [MPS] Do not dispatch empty job in  `bitwise_not` (#87286) (#87427)- 385022c76e4229a9393d63871225e0961e91c2db [ci] handle libomp upgrade on github (#87382) (#87408)- 3ffddc0b2d5fc963463c98b518b0eb1f88788a74 [functorch] fix AOTAutograd tutorial (#87415) (#87434)+ 55c76baf579cb6593f87d1a23e9a49afeb55f15a [1.13] Fix functorch version docs (#87392)- 59686b4c60289188436ac61576fd389821858fa5 [maskedtensor] fix docs formatting (#87387) (#87406)+ 0c0df0be7497112022804df03aeeb0fcbadc9243 Add `weights_only` option to `torch.load` (#87443)+ d253eb29d86af51ae17b950825ffdb5661b5af7f Avoid calling logging.basicConfig (#86959) (#87455)- d3aecbd9bc58a42366abffa63748d73c872bd927 Delete torch::deploy from pytorch core (#85953) (#85953) (#87454)- 51fa4fae41367a17f49ce648c7cdd6aa72f6e6ac Move PadNd from ATen/native to ATen (#87456)- f6c42ae2c29ba788523149dbcc791bf14530f93d Reenable `isinstance` with `torch.distributed.ReduceOp` (#87303) (#87463)- 6a8be2cb630ce79d654a707e5ea454d013acbda1 [ONNX] Reland: Update training state logic to support ScriptedModule (#86745) (#87457)+ 8569a44f38ff906103f40c6925dec366c7557943 [MPS] Revamp copy_to_mps_ implementation (#87475)- fdb18da4cc3ee04c2423eb33b2780edc4cae64e0 Fix distributed issue by including distributed files (#87612)- 341c377c0f595ff4e4a0fbdef21978c038b64b98 Add General Project Policies (#87385) (#87613)- 4e1a4b150a4ccd5f401a4293a0336a9a8a1dad2d fix docs push (#87498) (#87628)- 7c98e70d44abc7a1aead68b6ea6c8adc8c554db5 attempted fix for nvrtc with lovelace (#87611) (#87618)- 74a9ca993bd79f8131829e9c946657fa9a1d05ef [JIT][Security] Do not blindly eval input string (#89189) (#89925)+ ae2fe4033cf3b17259b17f351020b988fa893f91 Update masked.rst (#89758) (#89923)- c13d400bffe90e16b96520bbc8a41a6f0c9cd584 Use the Python frame safely in _pythonCallstack
2022-12-01T10:04:40.1418667Z GITHUB_REF_NAME=89997/merge
2022-12-01T10:04:40.1419017Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla
2022-12-01T10:04:40.1419287Z GITHUB_JOB=test
2022-12-01T10:04:40.1419477Z GITHUB_REPOSITORY=pytorch/pytorch
2022-12-01T10:04:40.1419771Z GITHUB_RETENTION_DAYS=90
2022-12-01T10:04:40.1419973Z OPENSSL_DIR=/opt/openssl
2022-12-01T10:04:40.1420161Z GITHUB_ACTION_REPOSITORY=
2022-12-01T10:04:40.1420675Z PATH=/opt/cache/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
2022-12-01T10:04:40.1421011Z GITHUB_BASE_REF=release/1.13
2022-12-01T10:04:40.1421196Z CI=true
2022-12-01T10:04:40.1421367Z GITHUB_REPOSITORY_OWNER=pytorch
2022-12-01T10:04:40.1421571Z INSTALLED_PROTOBUF=yes
2022-12-01T10:04:40.1421827Z GITHUB_HEAD_REF=release/1.13-callstack
2022-12-01T10:04:40.1422015Z GITHUB_ACTION_REF=
2022-12-01T10:04:40.1422295Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2
2022-12-01T10:04:40.1422532Z GITHUB_WORKFLOW=pull
2022-12-01T10:04:40.1422722Z DEBIAN_FRONTEND=noninteractive
2022-12-01T10:04:40.1423152Z GITHUB_OUTPUT=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_output_bc9efd0f-a7ec-45e6-b0aa-46e5f907846d
2022-12-01T10:04:40.1424235Z PR_BODY=Link to landed master PR (if applicable):https://github.com/pytorch/pytorch/pull/88993Criteria category:1: This prevents a crash, which was introduced [here](https://github.com/pytorch/pytorch/commit/4b7de265569f7fd731dd1cfea83ce804cc22f7c0#diff-45b117ca26b4fc9174fbe0d7a9cd8cb1c43964cd5e2bb20c7778ee00a942ef63), tagged for 1.132: Prevents a crashIm hoping this is a low-risk change, since its just changing one method for its safer form.
2022-12-01T10:04:40.1424866Z _=/usr/bin/env
2022-12-01T10:04:40.1425064Z + echo 'Testing pytorch'
2022-12-01T10:04:40.1425254Z Testing pytorch
2022-12-01T10:04:40.1425451Z + export LANG=C.UTF-8
2022-12-01T10:04:40.1425642Z + LANG=C.UTF-8
2022-12-01T10:04:40.1425802Z + PR_NUMBER=89997
2022-12-01T10:04:40.1425996Z + [[ distributed == \d\e\f\a\u\l\t ]]
2022-12-01T10:04:40.1426220Z + [[ distributed == \d\i\s\t\r\i\b\u\t\e\d ]]
2022-12-01T10:04:40.1426478Z + [[ linux-focal-py3.7-gcc7 == *rocm* ]]
2022-12-01T10:04:40.1426762Z + [[ distributed == \s\l\o\w ]]
2022-12-01T10:04:40.1427044Z + [[ linux-focal-py3.7-gcc7 == *slow-gradcheck* ]]
2022-12-01T10:04:40.1427365Z + [[ linux-focal-py3.7-gcc7 == *cuda* ]]
2022-12-01T10:04:40.1427638Z + [[ linux-focal-py3.7-gcc7 == *rocm* ]]
2022-12-01T10:04:40.1427912Z + [[ linux-focal-py3.7-gcc7 == *cuda11* ]]
2022-12-01T10:04:40.1428123Z + [[ distributed == *crossref* ]]
2022-12-01T10:04:40.1428327Z + [[ distributed == *dynamo* ]]
2022-12-01T10:04:40.1428538Z + [[ -n 89997 ]]
2022-12-01T10:04:40.1428742Z + [[ -z '' ]]
2022-12-01T10:04:40.1428953Z + export PYTORCH_TEST_SKIP_CUDA_MEM_LEAK_CHECK=1
2022-12-01T10:04:40.1429192Z + PYTORCH_TEST_SKIP_CUDA_MEM_LEAK_CHECK=1
2022-12-01T10:04:40.1429448Z + [[ linux-focal-py3.7-gcc7 == *rocm* ]]
2022-12-01T10:04:40.1429728Z + [[ linux-focal-py3.7-gcc7 != *-bazel-* ]]
2022-12-01T10:04:40.1429977Z + pip_install --user ninja
2022-12-01T10:04:40.1430231Z + pip install --progress-bar off --user ninja
2022-12-01T10:04:40.5645615Z Collecting ninja
2022-12-01T10:04:40.5812478Z   Downloading ninja-1.11.1-py2.py3-none-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (145 kB)
2022-12-01T10:04:41.3074067Z Installing collected packages: ninja
2022-12-01T10:04:41.3172548Z   WARNING: The script ninja is installed in '/var/lib/jenkins/.local/bin' which is not on PATH.
2022-12-01T10:04:41.3173175Z   Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
2022-12-01T10:04:41.3225286Z Successfully installed ninja-1.11.1
2022-12-01T10:04:41.4028668Z + export PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
2022-12-01T10:04:41.4029544Z + PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
2022-12-01T10:04:41.4030325Z + [[ linux-focal-py3.7-gcc7 == *asan* ]]
2022-12-01T10:04:41.4030577Z + [[ distributed == \n\o\g\p\u\_\N\O\_\A\V\X\2 ]]
2022-12-01T10:04:41.4030826Z + [[ distributed == \n\o\g\p\u\_\A\V\X\5\1\2 ]]
2022-12-01T10:04:41.4033630Z + [[ linux-focal-py3.7-gcc7 == *tbb* ]]
2022-12-01T10:04:41.4045373Z + [[ linux-focal-py3.7-gcc7 == *libtorch* ]]
2022-12-01T10:04:41.4045732Z + [[ linux-focal-py3.7-gcc7 == *-bazel-* ]]
2022-12-01T10:04:41.4048013Z + cd test
2022-12-01T10:04:41.4048535Z + python -c 'import torch; print(torch.__config__.show())'
2022-12-01T10:04:42.1915004Z PyTorch built with:
2022-12-01T10:04:42.1915447Z   - GCC 7.5
2022-12-01T10:04:42.1915672Z   - C++ Version: 201402
2022-12-01T10:04:42.1916325Z   - Intel(R) oneAPI Math Kernel Library Version 2022.0-Product Build 20211112 for Intel(R) 64 architecture applications
2022-12-01T10:04:42.1917090Z   - Intel(R) MKL-DNN v2.6.0 (Git Hash 52b5f107dd9cf10910aaa19cb47f3abf9b349815)
2022-12-01T10:04:42.1917440Z   - OpenMP 201511 (a.k.a. OpenMP 4.5)
2022-12-01T10:04:42.1917767Z   - LAPACK is enabled (usually provided by MKL)
2022-12-01T10:04:42.1918161Z   - NNPACK is enabled
2022-12-01T10:04:42.1918447Z   - CPU capability usage: AVX2
2022-12-01T10:04:42.1921597Z   - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CXX_COMPILER=/opt/cache/bin/c++, CXX_FLAGS= -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fopenmp -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOCUPTI -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -DEDGE_PROFILER_USE_KINETO -O2 -fPIC -Wno-narrowing -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wunused-local-typedefs -Wno-unused-parameter -Wno-unused-function -Wno-unused-result -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Werror -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=1.13.0, USE_CUDA=OFF, USE_CUDNN=OFF, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=OFF, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, 
2022-12-01T10:04:42.1923273Z 
2022-12-01T10:04:42.3551951Z + cd test
2022-12-01T10:04:42.3552605Z + python -c 'import torch; print(torch.__config__.parallel_info())'
2022-12-01T10:04:43.1216007Z ATen/Parallel:
2022-12-01T10:04:43.1216456Z 	at::get_num_threads() : 4
2022-12-01T10:04:43.1216840Z 	at::get_num_interop_threads() : 4
2022-12-01T10:04:43.1217146Z OpenMP 201511 (a.k.a. OpenMP 4.5)
2022-12-01T10:04:43.1217351Z 	omp_get_max_threads() : 4
2022-12-01T10:04:43.1217872Z Intel(R) oneAPI Math Kernel Library Version 2022.0-Product Build 20211112 for Intel(R) 64 architecture applications
2022-12-01T10:04:43.1218156Z 	mkl_get_max_threads() : 4
2022-12-01T10:04:43.1218505Z Intel(R) MKL-DNN v2.6.0 (Git Hash 52b5f107dd9cf10910aaa19cb47f3abf9b349815)
2022-12-01T10:04:43.1218788Z std::thread::hardware_concurrency() : 8
2022-12-01T10:04:43.1218985Z Environment variables:
2022-12-01T10:04:43.1219182Z 	OMP_NUM_THREADS : [not set]
2022-12-01T10:04:43.1219377Z 	MKL_NUM_THREADS : [not set]
2022-12-01T10:04:43.1219639Z ATen parallel backend: OpenMP
2022-12-01T10:04:43.1219829Z 
2022-12-01T10:04:43.2822669Z + [[ distributed == *backward* ]]
2022-12-01T10:04:43.2823102Z + [[ distributed == *xla* ]]
2022-12-01T10:04:43.2823445Z + [[ distributed == \j\i\t\_\l\e\g\a\c\y ]]
2022-12-01T10:04:43.2823866Z + [[ linux-focal-py3.7-gcc7 == *libtorch* ]]
2022-12-01T10:04:43.2824086Z + [[ distributed == distributed ]]
2022-12-01T10:04:43.2824290Z + install_torchdynamo
2022-12-01T10:04:43.2824470Z + local commit
2022-12-01T10:04:43.2826314Z ++ get_pinned_commit torchdynamo
2022-12-01T10:04:43.2826588Z ++ cat .github/ci_commit_pins/torchdynamo.txt
2022-12-01T10:04:43.2837417Z + commit=6ead5cae0d1234aa64db06fe230ef56e12ec76fe
2022-12-01T10:04:43.2838291Z + pip_install --user git+https://github.com/pytorch/torchdynamo.git@6ead5cae0d1234aa64db06fe230ef56e12ec76fe
2022-12-01T10:04:43.2839240Z + pip install --progress-bar off --user git+https://github.com/pytorch/torchdynamo.git@6ead5cae0d1234aa64db06fe230ef56e12ec76fe
2022-12-01T10:04:43.6425568Z Collecting git+https://github.com/pytorch/torchdynamo.git@6ead5cae0d1234aa64db06fe230ef56e12ec76fe
2022-12-01T10:04:43.6433108Z   Cloning https://github.com/pytorch/torchdynamo.git (to revision 6ead5cae0d1234aa64db06fe230ef56e12ec76fe) to /tmp/pip-req-build-na48yj0k
2022-12-01T10:04:43.6468171Z   Running command git clone --filter=blob:none --quiet https://github.com/pytorch/torchdynamo.git /tmp/pip-req-build-na48yj0k
2022-12-01T10:04:44.1143061Z   Running command git rev-parse -q --verify 'sha^6ead5cae0d1234aa64db06fe230ef56e12ec76fe'
2022-12-01T10:04:44.1177925Z   Running command git fetch -q https://github.com/pytorch/torchdynamo.git 6ead5cae0d1234aa64db06fe230ef56e12ec76fe
2022-12-01T10:04:44.3387094Z   Running command git checkout -q 6ead5cae0d1234aa64db06fe230ef56e12ec76fe
2022-12-01T10:04:44.7874964Z   Resolved https://github.com/pytorch/torchdynamo.git to commit 6ead5cae0d1234aa64db06fe230ef56e12ec76fe
2022-12-01T10:04:46.2671918Z   Preparing metadata (setup.py) ... [?25l- done
2022-12-01T10:04:46.2741946Z [?25hRequirement already satisfied: torch>=1.12.0 in /opt/conda/lib/python3.7/site-packages (from torchdynamo==1.13.0.dev0) (1.13.0a0+gitc13d400)
2022-12-01T10:04:46.2745766Z Requirement already satisfied: numpy in /opt/conda/lib/python3.7/site-packages (from torchdynamo==1.13.0.dev0) (1.18.5)
2022-12-01T10:04:46.3080039Z Collecting tabulate
2022-12-01T10:04:46.3262133Z   Downloading tabulate-0.9.0-py3-none-any.whl (35 kB)
2022-12-01T10:04:46.3314503Z Requirement already satisfied: pyyaml in /opt/conda/lib/python3.7/site-packages (from torchdynamo==1.13.0.dev0) (6.0)
2022-12-01T10:04:46.3318204Z Requirement already satisfied: sympy in /opt/conda/lib/python3.7/site-packages (from torchdynamo==1.13.0.dev0) (1.10.1)
2022-12-01T10:04:46.3351278Z Requirement already satisfied: typing-extensions in /opt/conda/lib/python3.7/site-packages (from torch>=1.12.0->torchdynamo==1.13.0.dev0) (4.3.0)
2022-12-01T10:04:46.3377078Z Requirement already satisfied: mpmath>=0.19 in /opt/conda/lib/python3.7/site-packages (from sympy->torchdynamo==1.13.0.dev0) (1.2.1)
2022-12-01T10:04:46.3470444Z Building wheels for collected packages: torchdynamo
2022-12-01T10:04:49.8896611Z   Building wheel for torchdynamo (setup.py) ... [?25l- \ | / - done
2022-12-01T10:04:49.8977537Z [?25h  Created wheel for torchdynamo: filename=torchdynamo-1.13.0.dev0-cp37-cp37m-linux_x86_64.whl size=2686267 sha256=13e39114c8b9b8a908cd1200512d72d2bc7d8b0d33bd17ae911e77251ca1c492
2022-12-01T10:04:49.8978556Z   Stored in directory: /var/lib/jenkins/.cache/pip/wheels/be/04/a5/adc64d091c88f64465d8ee6f21d5717177ec59bba7342b9732
2022-12-01T10:04:49.9001842Z Successfully built torchdynamo
2022-12-01T10:04:50.6214373Z Installing collected packages: tabulate, torchdynamo
2022-12-01T10:04:50.8927699Z Successfully installed tabulate-0.9.0 torchdynamo-1.13.0.dev0
2022-12-01T10:04:50.9885642Z + test_distributed
2022-12-01T10:04:50.9886169Z + echo 'Testing distributed python tests'
2022-12-01T10:04:50.9886411Z Testing distributed python tests
2022-12-01T10:04:50.9886760Z + python test/run_test.py --distributed-tests --shard 2 2 --verbose
2022-12-01T10:04:52.1588109Z No CUDA runtime is found, using CUDA_HOME='/usr/local/cuda'
2022-12-01T10:04:52.2235936Z Ignoring disabled issues:  []
2022-12-01T10:04:52.2374763Z Found test time stats from artifacts
2022-12-01T10:04:52.2382983Z Selected tests:
2022-12-01T10:04:52.2383227Z  distributed/test_distributed_spawn
2022-12-01T10:04:52.2383577Z  distributed/test_c10d_gloo
2022-12-01T10:04:52.2383948Z  distributed/rpc/cuda/test_tensorpipe_agent
2022-12-01T10:04:52.2384313Z  distributed/fsdp/test_fsdp_state_dict
2022-12-01T10:04:52.2384686Z  distributed/fsdp/test_fsdp_optim_state
2022-12-01T10:04:52.2385093Z  distributed/fsdp/test_fsdp_mixed_precision
2022-12-01T10:04:52.2385479Z  distributed/fsdp/test_wrap
2022-12-01T10:04:52.2385870Z  distributed/optim/test_zero_redundancy_optimizer
2022-12-01T10:04:52.2386317Z  distributed/fsdp/test_fsdp_comm_hooks
2022-12-01T10:04:52.2386725Z  distributed/fsdp/test_fsdp_grad_acc
2022-12-01T10:04:52.2387136Z  distributed/elastic/multiprocessing/api_test
2022-12-01T10:04:52.2387386Z  distributed/test_pg_wrapper
2022-12-01T10:04:52.2387614Z  distributed/fsdp/test_fsdp_comm
2022-12-01T10:04:52.2387968Z  distributed/fsdp/test_fsdp_sharded_grad_scaler
2022-12-01T10:04:52.2388263Z  distributed/test_c10d_common
2022-12-01T10:04:52.2388654Z  distributed/elastic/timer/local_timer_example
2022-12-01T10:04:52.2389086Z  distributed/fsdp/test_fsdp_tp_integration
2022-12-01T10:04:52.2389498Z  distributed/algorithms/quantization/test_quantization
2022-12-01T10:04:52.2389963Z  distributed/fsdp/test_fsdp_clip_grad_norm
2022-12-01T10:04:52.2390356Z  distributed/_shard/sharded_tensor/test_sharded_tensor
2022-12-01T10:04:52.2390600Z  distributed/fsdp/test_distributed_checkpoint
2022-12-01T10:04:52.2390846Z  distributed/elastic/utils/distributed_test
2022-12-01T10:04:52.2391073Z  distributed/fsdp/test_fsdp_uneven
2022-12-01T10:04:52.2391280Z  distributed/rpc/test_share_memory
2022-12-01T10:04:52.2391511Z  distributed/fsdp/test_fsdp_multiple_forward
2022-12-01T10:04:52.2391741Z  distributed/fsdp/test_fsdp_overlap
2022-12-01T10:04:52.2391955Z  distributed/fsdp/test_checkpoint_wrapper
2022-12-01T10:04:52.2392185Z  distributed/elastic/utils/logging_test
2022-12-01T10:04:52.2392408Z  distributed/nn/jit/test_instantiator
2022-12-01T10:04:52.2392605Z  distributed/fsdp/test_utils
2022-12-01T10:04:52.2392809Z  distributed/test_data_parallel
2022-12-01T10:04:52.2393047Z  distributed/_shard/sharded_tensor/ops/test_matrix_ops
2022-12-01T10:04:52.2393292Z  distributed/_shard/sharded_tensor/ops/test_linear
2022-12-01T10:04:52.2393559Z  distributed/_shard/sharded_tensor/ops/test_elementwise_ops
2022-12-01T10:04:52.2394045Z  distributed/algorithms/ddp_comm_hooks/test_ddp_hooks
2022-12-01T10:04:52.2394349Z  distributed/test_c10d_spawn_nccl
2022-12-01T10:04:52.2394585Z  distributed/_shard/sharded_tensor/ops/test_chunk
2022-12-01T10:04:52.2394851Z  distributed/_shard/sharded_tensor/test_megatron_prototype
2022-12-01T10:04:52.2395120Z  distributed/_shard/sharded_tensor/ops/test_binary_cmp
2022-12-01T10:04:52.2395377Z  distributed/_shard/checkpoint/test_file_system_checkpoint
2022-12-01T10:04:52.2395631Z  distributed/_shard/checkpoint/test_planner
2022-12-01T10:04:52.2395881Z  distributed/_shard/sharded_tensor/ops/test_math_ops
2022-12-01T10:04:52.2396117Z  distributed/_shard/test_sharder
2022-12-01T10:04:52.2396333Z  distributed/elastic/timer/api_test
2022-12-01T10:04:52.2396568Z  distributed/fsdp/test_fsdp_param_exec_order_wrap
2022-12-01T10:04:52.2396799Z  distributed/pipeline/sync/skip/test_api
2022-12-01T10:04:52.2397057Z  distributed/pipeline/sync/skip/test_inspect_skip_layout
2022-12-01T10:04:52.2397317Z  distributed/pipeline/sync/skip/test_portal
2022-12-01T10:04:52.2397559Z  distributed/pipeline/sync/skip/test_tracker
2022-12-01T10:04:52.2397783Z  distributed/pipeline/sync/test_balance
2022-12-01T10:04:52.2398016Z  distributed/pipeline/sync/test_checkpoint
2022-12-01T10:04:52.2398269Z  distributed/pipeline/sync/test_deferred_batch_norm
2022-12-01T10:04:52.2398495Z  distributed/pipeline/sync/test_inplace
2022-12-01T10:04:52.2398722Z  distributed/pipeline/sync/test_phony
2022-12-01T10:04:52.2398958Z  distributed/pipeline/sync/test_pipeline
2022-12-01T10:04:52.2399189Z  distributed/pipeline/sync/test_transparency
2022-12-01T10:04:52.2399401Z  distributed/test_nccl
2022-12-01T10:04:52.4048653Z Prioritized test from test file changes.
2022-12-01T10:04:52.4048994Z reordering tests for PR:
2022-12-01T10:04:52.4051062Z prioritized: ['distributed/test_c10d_gloo', 'distributed/fsdp/test_fsdp_state_dict', 'distributed/fsdp/test_fsdp_optim_state', 'distributed/fsdp/test_fsdp_mixed_precision', 'distributed/fsdp/test_wrap', 'distributed/fsdp/test_fsdp_comm_hooks', 'distributed/fsdp/test_fsdp_grad_acc', 'distributed/fsdp/test_fsdp_comm', 'distributed/fsdp/test_fsdp_sharded_grad_scaler', 'distributed/test_c10d_common', 'distributed/fsdp/test_fsdp_tp_integration', 'distributed/fsdp/test_fsdp_clip_grad_norm', 'distributed/fsdp/test_distributed_checkpoint', 'distributed/fsdp/test_fsdp_uneven', 'distributed/fsdp/test_fsdp_multiple_forward', 'distributed/fsdp/test_fsdp_overlap', 'distributed/fsdp/test_checkpoint_wrapper', 'distributed/fsdp/test_utils', 'distributed/algorithms/ddp_comm_hooks/test_ddp_hooks', 'distributed/_shard/checkpoint/test_file_system_checkpoint', 'distributed/_shard/checkpoint/test_planner', 'distributed/fsdp/test_fsdp_param_exec_order_wrap']
2022-12-01T10:04:52.4055085Z the rest: ['distributed/test_distributed_spawn', 'distributed/rpc/cuda/test_tensorpipe_agent', 'distributed/optim/test_zero_redundancy_optimizer', 'distributed/elastic/multiprocessing/api_test', 'distributed/test_pg_wrapper', 'distributed/elastic/timer/local_timer_example', 'distributed/algorithms/quantization/test_quantization', 'distributed/_shard/sharded_tensor/test_sharded_tensor', 'distributed/elastic/utils/distributed_test', 'distributed/rpc/test_share_memory', 'distributed/elastic/utils/logging_test', 'distributed/nn/jit/test_instantiator', 'distributed/test_data_parallel', 'distributed/_shard/sharded_tensor/ops/test_matrix_ops', 'distributed/_shard/sharded_tensor/ops/test_linear', 'distributed/_shard/sharded_tensor/ops/test_elementwise_ops', 'distributed/test_c10d_spawn_nccl', 'distributed/_shard/sharded_tensor/ops/test_chunk', 'distributed/_shard/sharded_tensor/test_megatron_prototype', 'distributed/_shard/sharded_tensor/ops/test_binary_cmp', 'distributed/_shard/sharded_tensor/ops/test_math_ops', 'distributed/_shard/test_sharder', 'distributed/elastic/timer/api_test', 'distributed/pipeline/sync/skip/test_api', 'distributed/pipeline/sync/skip/test_inspect_skip_layout', 'distributed/pipeline/sync/skip/test_portal', 'distributed/pipeline/sync/skip/test_tracker', 'distributed/pipeline/sync/test_balance', 'distributed/pipeline/sync/test_checkpoint', 'distributed/pipeline/sync/test_deferred_batch_norm', 'distributed/pipeline/sync/test_inplace', 'distributed/pipeline/sync/test_phony', 'distributed/pipeline/sync/test_pipeline', 'distributed/pipeline/sync/test_transparency', 'distributed/test_nccl']
2022-12-01T10:04:52.4056984Z 
2022-12-01T10:04:52.4057469Z Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/slow-tests.json to /var/lib/jenkins/workspace/test/.pytorch-slow-tests.json
2022-12-01T10:04:52.4302321Z Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/disabled-tests-condensed.json to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json
2022-12-01T10:04:52.4450220Z parallel (file granularity) tests:
2022-12-01T10:04:52.4450674Z  
2022-12-01T10:04:52.4451004Z serial (file granularity) tests:
2022-12-01T10:04:52.4451377Z  distributed/test_c10d_gloo
2022-12-01T10:04:52.4451748Z  distributed/fsdp/test_fsdp_state_dict
2022-12-01T10:04:52.4452178Z  distributed/fsdp/test_fsdp_optim_state
2022-12-01T10:04:52.4452569Z  distributed/fsdp/test_fsdp_mixed_precision
2022-12-01T10:04:52.4452885Z  distributed/fsdp/test_wrap
2022-12-01T10:04:52.4453272Z  distributed/fsdp/test_fsdp_comm_hooks
2022-12-01T10:04:52.4453696Z  distributed/fsdp/test_fsdp_grad_acc
2022-12-01T10:04:52.4453941Z  distributed/fsdp/test_fsdp_comm
2022-12-01T10:04:52.4454227Z  distributed/fsdp/test_fsdp_sharded_grad_scaler
2022-12-01T10:04:52.4454606Z  distributed/test_c10d_common
2022-12-01T10:04:52.4455008Z  distributed/fsdp/test_fsdp_tp_integration
2022-12-01T10:04:52.4455415Z  distributed/fsdp/test_fsdp_clip_grad_norm
2022-12-01T10:04:52.4455661Z  distributed/fsdp/test_distributed_checkpoint
2022-12-01T10:04:52.4455892Z  distributed/fsdp/test_fsdp_uneven
2022-12-01T10:04:52.4456114Z  distributed/fsdp/test_fsdp_multiple_forward
2022-12-01T10:04:52.4456343Z  distributed/fsdp/test_fsdp_overlap
2022-12-01T10:04:52.4456577Z  distributed/fsdp/test_checkpoint_wrapper
2022-12-01T10:04:52.4456783Z  distributed/fsdp/test_utils
2022-12-01T10:04:52.4457025Z  distributed/algorithms/ddp_comm_hooks/test_ddp_hooks
2022-12-01T10:04:52.4457297Z  distributed/_shard/checkpoint/test_file_system_checkpoint
2022-12-01T10:04:52.4457542Z  distributed/_shard/checkpoint/test_planner
2022-12-01T10:04:52.4457785Z  distributed/fsdp/test_fsdp_param_exec_order_wrap
2022-12-01T10:04:52.4458018Z  distributed/test_distributed_spawn
2022-12-01T10:04:52.4458236Z  distributed/rpc/cuda/test_tensorpipe_agent
2022-12-01T10:04:52.4458479Z  distributed/optim/test_zero_redundancy_optimizer
2022-12-01T10:04:52.4458749Z  distributed/elastic/multiprocessing/api_test
2022-12-01T10:04:52.4458973Z  distributed/test_pg_wrapper
2022-12-01T10:04:52.4459185Z  distributed/elastic/timer/local_timer_example
2022-12-01T10:04:52.4459440Z  distributed/algorithms/quantization/test_quantization
2022-12-01T10:04:52.4459788Z  distributed/_shard/sharded_tensor/test_sharded_tensor
2022-12-01T10:04:52.4460029Z  distributed/elastic/utils/distributed_test
2022-12-01T10:04:52.4460261Z  distributed/rpc/test_share_memory
2022-12-01T10:04:52.4460685Z  distributed/elastic/utils/logging_test
2022-12-01T10:04:52.4460899Z  distributed/nn/jit/test_instantiator
2022-12-01T10:04:52.4461118Z  distributed/test_data_parallel
2022-12-01T10:04:52.4461358Z  distributed/_shard/sharded_tensor/ops/test_matrix_ops
2022-12-01T10:04:52.4461603Z  distributed/_shard/sharded_tensor/ops/test_linear
2022-12-01T10:04:52.4461872Z  distributed/_shard/sharded_tensor/ops/test_elementwise_ops
2022-12-01T10:04:52.4462114Z  distributed/test_c10d_spawn_nccl
2022-12-01T10:04:52.4462347Z  distributed/_shard/sharded_tensor/ops/test_chunk
2022-12-01T10:04:52.4462599Z  distributed/_shard/sharded_tensor/test_megatron_prototype
2022-12-01T10:04:52.4462866Z  distributed/_shard/sharded_tensor/ops/test_binary_cmp
2022-12-01T10:04:52.4463124Z  distributed/_shard/sharded_tensor/ops/test_math_ops
2022-12-01T10:04:52.4463490Z  distributed/_shard/test_sharder
2022-12-01T10:04:52.4463710Z  distributed/elastic/timer/api_test
2022-12-01T10:04:52.4464015Z  distributed/pipeline/sync/skip/test_api
2022-12-01T10:04:52.4464263Z  distributed/pipeline/sync/skip/test_inspect_skip_layout
2022-12-01T10:04:52.4464523Z  distributed/pipeline/sync/skip/test_portal
2022-12-01T10:04:52.4464767Z  distributed/pipeline/sync/skip/test_tracker
2022-12-01T10:04:52.4464993Z  distributed/pipeline/sync/test_balance
2022-12-01T10:04:52.4465231Z  distributed/pipeline/sync/test_checkpoint
2022-12-01T10:04:52.4465483Z  distributed/pipeline/sync/test_deferred_batch_norm
2022-12-01T10:04:52.4465712Z  distributed/pipeline/sync/test_inplace
2022-12-01T10:04:52.4465942Z  distributed/pipeline/sync/test_phony
2022-12-01T10:04:52.4466174Z  distributed/pipeline/sync/test_pipeline
2022-12-01T10:04:52.4466415Z  distributed/pipeline/sync/test_transparency
2022-12-01T10:04:52.4466620Z  distributed/test_nccl
2022-12-01T10:04:53.6111815Z No CUDA runtime is found, using CUDA_HOME='/usr/local/cuda'
2022-12-01T10:04:53.6172161Z No CUDA runtime is found, using CUDA_HOME='/usr/local/cuda'
2022-12-01T10:04:53.6754509Z Ignoring disabled issues:  []
2022-12-01T10:04:53.6806924Z Ignoring disabled issues:  []
2022-12-01T10:04:53.9877293Z Running distributed/test_c10d_gloo ... [2022-12-01 10:04:53.987340]
2022-12-01T10:04:53.9880034Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_c10d_gloo.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-12-01 10:04:53.987753]
2022-12-01T10:11:41.9310576Z 
2022-12-01T10:11:41.9311163Z Expand the folded group to see the log file of distributed/test_c10d_gloo
2022-12-01T10:11:41.9317757Z ##[group]PRINTING LOG FILE of distributed/test_c10d_gloo (/var/lib/jenkins/workspace/test/test-reports/distributed-test_c10d_gloo_zqx1eqdr)
2022-12-01T10:11:41.9319512Z <unittest.suite.TestSuite tests=[<__main__.CommTest testMethod=test_broadcast_coalesced_gloo_cpu>, <__main__.CommTest testMethod=test_broadcast_coalesced_gloo_cuda>, <__main__.CommTest testMethod=test_gloo_barrier_device_ids>, <__main__.CommTest testMethod=test_gloo_rank_membership>, <__main__.CommTest testMethod=test_gloo_warn_not_in_group>, <__main__.CommTest testMethod=test_sequence_num_incremented_gloo_default>, <__main__.CommTest testMethod=test_sequence_num_incremented_gloo_subgroup>, <__main__.CommTest testMethod=test_sequence_num_set_default_pg_gloo>, <__main__.CommTest testMethod=test_sequence_num_set_gloo_new_group>, <__main__.CommTest testMethod=test_tensor_dtype_complex>, <__main__.CommTest testMethod=test_tensor_dtype_mismatch>]>
2022-12-01T10:11:41.9321063Z test_broadcast_coalesced_gloo_cpu (__main__.CommTest)
2022-12-01T10:11:41.9321491Z test_broadcast_coalesced_gloo_cuda (__main__.CommTest)
2022-12-01T10:11:41.9321893Z test_gloo_barrier_device_ids (__main__.CommTest)
2022-12-01T10:11:41.9322279Z test_gloo_rank_membership (__main__.CommTest)
2022-12-01T10:11:41.9322678Z test_gloo_warn_not_in_group (__main__.CommTest)
2022-12-01T10:11:41.9323090Z test_sequence_num_incremented_gloo_default (__main__.CommTest)
2022-12-01T10:11:41.9323515Z test_sequence_num_incremented_gloo_subgroup (__main__.CommTest)
2022-12-01T10:11:41.9323934Z test_sequence_num_set_default_pg_gloo (__main__.CommTest)
2022-12-01T10:11:41.9329109Z test_sequence_num_set_gloo_new_group (__main__.CommTest)
2022-12-01T10:11:41.9329530Z test_tensor_dtype_complex (__main__.CommTest)
2022-12-01T10:11:41.9329939Z test_tensor_dtype_mismatch (__main__.CommTest)
2022-12-01T10:11:41.9331467Z <unittest.suite.TestSuite tests=[<__main__.CompilerTest testMethod=test_allgather_work_wait_cpu>, <__main__.CompilerTest testMethod=test_allgather_work_wait_gpu>, <__main__.CompilerTest testMethod=test_allreduce_work_wait_cpu>, <__main__.CompilerTest testMethod=test_allreduce_work_wait_gpu>, <__main__.CompilerTest testMethod=test_broadcast_work_wait_cpu>, <__main__.CompilerTest testMethod=test_broadcast_work_wait_gpu>, <__main__.CompilerTest testMethod=test_consecutive_comm_work_wait_cpu>, <__main__.CompilerTest testMethod=test_consecutive_comm_work_wait_gpu>, <__main__.CompilerTest testMethod=test_nested_comm_tensor_wrapping>, <__main__.CompilerTest testMethod=test_scatter_work_wait_cpu>, <__main__.CompilerTest testMethod=test_scatter_work_wait_gpu>]>
2022-12-01T10:11:41.9335771Z test_allgather_work_wait_cpu (__main__.CompilerTest)
2022-12-01T10:11:41.9336166Z test_allgather_work_wait_gpu (__main__.CompilerTest)
2022-12-01T10:11:41.9336652Z test_allreduce_work_wait_cpu (__main__.CompilerTest)
2022-12-01T10:11:41.9337114Z test_allreduce_work_wait_gpu (__main__.CompilerTest)
2022-12-01T10:11:41.9337390Z test_broadcast_work_wait_cpu (__main__.CompilerTest)
2022-12-01T10:11:41.9337711Z test_broadcast_work_wait_gpu (__main__.CompilerTest)
2022-12-01T10:11:41.9338180Z test_consecutive_comm_work_wait_cpu (__main__.CompilerTest)
2022-12-01T10:11:41.9338655Z test_consecutive_comm_work_wait_gpu (__main__.CompilerTest)
2022-12-01T10:11:41.9339053Z test_nested_comm_tensor_wrapping (__main__.CompilerTest)
2022-12-01T10:11:41.9339405Z test_scatter_work_wait_cpu (__main__.CompilerTest)
2022-12-01T10:11:41.9339782Z test_scatter_work_wait_gpu (__main__.CompilerTest)
2022-12-01T10:11:41.9346567Z <unittest.suite.TestSuite tests=[<__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_dynamic_module>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_dynamic_weight_sharing>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_once_use_reentrant_False>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_once_use_reentrant_True>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_twice_static_graph_use_reentrant_False>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_twice_static_graph_use_reentrant_True>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_twice_use_reentrant_False>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_twice_use_reentrant_True>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_twice_weight_sharing>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_unused_params_use_reentrant_False>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_unused_params_use_reentrant_True>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_weight_sharing_use_reentrant_False>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_weight_sharing_use_reentrant_True>, <__main__.DistributedDataParallelTest testMethod=test_ddp_comm_hook_future_passing_cpu>, <__main__.DistributedDataParallelTest testMethod=test_ddp_comm_hook_future_passing_gpu_gloo>, <__main__.DistributedDataParallelTest testMethod=test_ddp_comm_hook_register_just_once>, <__main__.DistributedDataParallelTest testMethod=test_ddp_comm_hook_sparse_gradients>, <__main__.DistributedDataParallelTest testMethod=test_ddp_invalid_comm_hook_init>, <__main__.DistributedDataParallelTest testMethod=test_ddp_invalid_comm_hook_return_type>, <__main__.DistributedDataParallelTest testMethod=test_find_unused_parameters_when_unused_parameters_empty>, <__main__.DistributedDataParallelTest testMethod=test_global_local_unused_params_grad>, <__main__.DistributedDataParallelTest testMethod=test_global_local_unused_params_grad_with_grad_is_view>, <__main__.DistributedDataParallelTest testMethod=test_global_local_unused_params_grad_with_static_graph>, <__main__.DistributedDataParallelTest testMethod=test_gloo_backend_1gpu_module_device_ids_integer_list>, <__main__.DistributedDataParallelTest testMethod=test_gloo_backend_1gpu_module_device_ids_torch_device_list>, <__main__.DistributedDataParallelTest testMethod=test_gloo_backend_2gpu_module>, <__main__.DistributedDataParallelTest testMethod=test_gloo_backend_4gpu_module>, <__main__.DistributedDataParallelTest testMethod=test_gloo_backend_cpu_module>, <__main__.DistributedDataParallelTest testMethod=test_gloo_backend_cpu_module_grad_is_view>, <__main__.DistributedDataParallelTest testMethod=test_ignored_output>, <__main__.DistributedDataParallelTest testMethod=test_ignored_output_with_unused_parameters>, <__main__.DistributedDataParallelTest testMethod=test_invalid_powerSGD_state>, <__main__.DistributedDataParallelTest testMethod=test_save_load_checkpoint>, <__main__.DistributedDataParallelTest testMethod=test_sparse_gradients>, <__main__.DistributedDataParallelTest testMethod=test_sparse_gradients_grad_is_view>, <__main__.DistributedDataParallelTest testMethod=test_sync_batch_norm_empty_input>, <__main__.DistributedDataParallelTest testMethod=test_sync_batch_norm_only_empty_input>]>
2022-12-01T10:11:41.9352778Z test_ddp_checkpointing_dynamic_module (__main__.DistributedDataParallelTest)
2022-12-01T10:11:41.9353275Z test_ddp_checkpointing_dynamic_weight_sharing (__main__.DistributedDataParallelTest)
2022-12-01T10:11:41.9353836Z test_ddp_checkpointing_once_use_reentrant_False (__main__.DistributedDataParallelTest)
2022-12-01T10:11:41.9354544Z test_ddp_checkpointing_once_use_reentrant_True (__main__.DistributedDataParallelTest)
2022-12-01T10:11:41.9355266Z test_ddp_checkpointing_twice_static_graph_use_reentrant_False (__main__.DistributedDataParallelTest)
2022-12-01T10:11:41.9355961Z test_ddp_checkpointing_twice_static_graph_use_reentrant_True (__main__.DistributedDataParallelTest)
2022-12-01T10:11:41.9356766Z test_ddp_checkpointing_twice_use_reentrant_False (__main__.DistributedDataParallelTest)
2022-12-01T10:11:41.9357559Z test_ddp_checkpointing_twice_use_reentrant_True (__main__.DistributedDataParallelTest)
2022-12-01T10:11:41.9358311Z test_ddp_checkpointing_twice_weight_sharing (__main__.DistributedDataParallelTest)
2022-12-01T10:11:41.9359114Z test_ddp_checkpointing_unused_params_use_reentrant_False (__main__.DistributedDataParallelTest)
2022-12-01T10:11:41.9359942Z test_ddp_checkpointing_unused_params_use_reentrant_True (__main__.DistributedDataParallelTest)
2022-12-01T10:11:41.9360776Z test_ddp_checkpointing_weight_sharing_use_reentrant_False (__main__.DistributedDataParallelTest)
2022-12-01T10:11:41.9361587Z test_ddp_checkpointing_weight_sharing_use_reentrant_True (__main__.DistributedDataParallelTest)
2022-12-01T10:11:41.9362362Z test_ddp_comm_hook_future_passing_cpu (__main__.DistributedDataParallelTest)
2022-12-01T10:11:41.9363098Z test_ddp_comm_hook_future_passing_gpu_gloo (__main__.DistributedDataParallelTest)
2022-12-01T10:11:41.9363830Z test_ddp_comm_hook_register_just_once (__main__.DistributedDataParallelTest)
2022-12-01T10:11:41.9364526Z test_ddp_comm_hook_sparse_gradients (__main__.DistributedDataParallelTest)
2022-12-01T10:11:41.9365228Z test_ddp_invalid_comm_hook_init (__main__.DistributedDataParallelTest)
2022-12-01T10:11:41.9365941Z test_ddp_invalid_comm_hook_return_type (__main__.DistributedDataParallelTest)
2022-12-01T10:11:41.9366705Z test_find_unused_parameters_when_unused_parameters_empty (__main__.DistributedDataParallelTest)
2022-12-01T10:11:41.9367478Z test_global_local_unused_params_grad (__main__.DistributedDataParallelTest)
2022-12-01T10:11:41.9368244Z test_global_local_unused_params_grad_with_grad_is_view (__main__.DistributedDataParallelTest)
2022-12-01T10:11:41.9369039Z test_global_local_unused_params_grad_with_static_graph (__main__.DistributedDataParallelTest)
2022-12-01T10:11:41.9369819Z test_gloo_backend_1gpu_module_device_ids_integer_list (__main__.DistributedDataParallelTest)
2022-12-01T10:11:41.9370620Z test_gloo_backend_1gpu_module_device_ids_torch_device_list (__main__.DistributedDataParallelTest)
2022-12-01T10:11:41.9371366Z test_gloo_backend_2gpu_module (__main__.DistributedDataParallelTest)
2022-12-01T10:11:41.9372026Z test_gloo_backend_4gpu_module (__main__.DistributedDataParallelTest)
2022-12-01T10:11:41.9372714Z test_gloo_backend_cpu_module (__main__.DistributedDataParallelTest)
2022-12-01T10:11:41.9373432Z test_gloo_backend_cpu_module_grad_is_view (__main__.DistributedDataParallelTest)
2022-12-01T10:11:41.9374115Z test_ignored_output (__main__.DistributedDataParallelTest)
2022-12-01T10:11:41.9374792Z test_ignored_output_with_unused_parameters (__main__.DistributedDataParallelTest)
2022-12-01T10:11:41.9375630Z test_invalid_powerSGD_state (__main__.DistributedDataParallelTest)
2022-12-01T10:11:41.9376368Z test_save_load_checkpoint (__main__.DistributedDataParallelTest)
2022-12-01T10:11:41.9377015Z test_sparse_gradients (__main__.DistributedDataParallelTest)
2022-12-01T10:11:41.9377706Z test_sparse_gradients_grad_is_view (__main__.DistributedDataParallelTest)
2022-12-01T10:11:41.9378530Z test_sync_batch_norm_empty_input (__main__.DistributedDataParallelTest)
2022-12-01T10:11:41.9379227Z test_sync_batch_norm_only_empty_input (__main__.DistributedDataParallelTest)
2022-12-01T10:11:41.9380161Z <unittest.suite.TestSuite tests=[<__main__.GlooProcessGroupWithDispatchedCollectivesTests testMethod=test_collectives>]>
2022-12-01T10:11:41.9381218Z test_collectives (__main__.GlooProcessGroupWithDispatchedCollectivesTests)
2022-12-01T10:11:41.9381900Z <unittest.suite.TestSuite tests=[]>
2022-12-01T10:11:41.9391012Z <unittest.suite.TestSuite tests=[<__main__.ProcessGroupGlooTest testMethod=test_allgather_basics>, <__main__.ProcessGroupGlooTest testMethod=test_allgather_basics_cuda>, <__main__.ProcessGroupGlooTest testMethod=test_allgather_checks>, <__main__.ProcessGroupGlooTest testMethod=test_allgather_coalesced_async>, <__main__.ProcessGroupGlooTest testMethod=test_allgather_coalesced_checks>, <__main__.ProcessGroupGlooTest testMethod=test_allgather_noncontiguous_input>, <__main__.ProcessGroupGlooTest testMethod=test_allgather_stress>, <__main__.ProcessGroupGlooTest testMethod=test_allgather_stress_cuda>, <__main__.ProcessGroupGlooTest testMethod=test_allreduce_basics>, <__main__.ProcessGroupGlooTest testMethod=test_allreduce_basics_cuda>, <__main__.ProcessGroupGlooTest testMethod=test_allreduce_basics_cuda_using_work_api>, <__main__.ProcessGroupGlooTest testMethod=test_allreduce_basics_using_work_api>, <__main__.ProcessGroupGlooTest testMethod=test_allreduce_checks>, <__main__.ProcessGroupGlooTest testMethod=test_allreduce_coalesced_async>, <__main__.ProcessGroupGlooTest testMethod=test_allreduce_coalesced_basics>, <__main__.ProcessGroupGlooTest testMethod=test_allreduce_coalesced_checks>, <__main__.ProcessGroupGlooTest testMethod=test_allreduce_coalesced_checks_cuda>, <__main__.ProcessGroupGlooTest testMethod=test_allreduce_coalesced_stress>, <__main__.ProcessGroupGlooTest testMethod=test_allreduce_stress>, <__main__.ProcessGroupGlooTest testMethod=test_allreduce_stress_cuda>, <__main__.ProcessGroupGlooTest testMethod=test_barrier_implies_wait>, <__main__.ProcessGroupGlooTest testMethod=test_broadcast_basics>, <__main__.ProcessGroupGlooTest testMethod=test_broadcast_basics_cuda>, <__main__.ProcessGroupGlooTest testMethod=test_broadcast_checks>, <__main__.ProcessGroupGlooTest testMethod=test_broadcast_stress>, <__main__.ProcessGroupGlooTest testMethod=test_broadcast_stress_cuda>, <__main__.ProcessGroupGlooTest testMethod=test_empty_tensors>, <__main__.ProcessGroupGlooTest testMethod=test_gather_basics>, <__main__.ProcessGroupGlooTest testMethod=test_gather_basics_cuda>, <__main__.ProcessGroupGlooTest testMethod=test_gather_checks>, <__main__.ProcessGroupGlooTest testMethod=test_gather_noncontiguous_input>, <__main__.ProcessGroupGlooTest testMethod=test_gather_stress>, <__main__.ProcessGroupGlooTest testMethod=test_gather_stress_cuda>, <__main__.ProcessGroupGlooTest testMethod=test_multi_device_constructor>, <__main__.ProcessGroupGlooTest testMethod=test_reduce_basics>, <__main__.ProcessGroupGlooTest testMethod=test_reduce_basics_cuda>, <__main__.ProcessGroupGlooTest testMethod=test_reduce_checks>, <__main__.ProcessGroupGlooTest testMethod=test_reduce_stress>, <__main__.ProcessGroupGlooTest testMethod=test_reduce_stress_cuda>, <__main__.ProcessGroupGlooTest testMethod=test_round_robin>, <__main__.ProcessGroupGlooTest testMethod=test_round_robin_create_destroy>, <__main__.ProcessGroupGlooTest testMethod=test_scatter_basics>, <__main__.ProcessGroupGlooTest testMethod=test_scatter_basics_cuda>, <__main__.ProcessGroupGlooTest testMethod=test_scatter_checks>, <__main__.ProcessGroupGlooTest testMethod=test_scatter_stress>, <__main__.ProcessGroupGlooTest testMethod=test_scatter_stress_cuda>, <__main__.ProcessGroupGlooTest testMethod=test_send_recv_all_to_all>, <__main__.ProcessGroupGlooTest testMethod=test_sparse_allreduce_basics>, <__main__.ProcessGroupGlooTest testMethod=test_sparse_allreduce_basics_cuda>, <__main__.ProcessGroupGlooTest testMethod=test_sparse_allreduce_checks>]>
2022-12-01T10:11:41.9398960Z test_allgather_basics (__main__.ProcessGroupGlooTest)
2022-12-01T10:11:41.9399412Z test_allgather_basics_cuda (__main__.ProcessGroupGlooTest)
2022-12-01T10:11:41.9399823Z test_allgather_checks (__main__.ProcessGroupGlooTest)
2022-12-01T10:11:41.9400277Z test_allgather_coalesced_async (__main__.ProcessGroupGlooTest)
2022-12-01T10:11:41.9400644Z test_allgather_coalesced_checks (__main__.ProcessGroupGlooTest)
2022-12-01T10:11:41.9400935Z test_allgather_noncontiguous_input (__main__.ProcessGroupGlooTest)
2022-12-01T10:11:41.9401226Z test_allgather_stress (__main__.ProcessGroupGlooTest)
2022-12-01T10:11:41.9401508Z test_allgather_stress_cuda (__main__.ProcessGroupGlooTest)
2022-12-01T10:11:41.9401777Z test_allreduce_basics (__main__.ProcessGroupGlooTest)
2022-12-01T10:11:41.9402056Z test_allreduce_basics_cuda (__main__.ProcessGroupGlooTest)
2022-12-01T10:11:41.9402397Z test_allreduce_basics_cuda_using_work_api (__main__.ProcessGroupGlooTest)
2022-12-01T10:11:41.9402930Z test_allreduce_basics_using_work_api (__main__.ProcessGroupGlooTest)
2022-12-01T10:11:41.9403349Z test_allreduce_checks (__main__.ProcessGroupGlooTest)
2022-12-01T10:11:41.9403771Z test_allreduce_coalesced_async (__main__.ProcessGroupGlooTest)
2022-12-01T10:11:41.9404237Z test_allreduce_coalesced_basics (__main__.ProcessGroupGlooTest)
2022-12-01T10:11:41.9404684Z test_allreduce_coalesced_checks (__main__.ProcessGroupGlooTest)
2022-12-01T10:11:41.9405160Z test_allreduce_coalesced_checks_cuda (__main__.ProcessGroupGlooTest)
2022-12-01T10:11:41.9405646Z test_allreduce_coalesced_stress (__main__.ProcessGroupGlooTest)
2022-12-01T10:11:41.9406094Z test_allreduce_stress (__main__.ProcessGroupGlooTest)
2022-12-01T10:11:41.9406531Z test_allreduce_stress_cuda (__main__.ProcessGroupGlooTest)
2022-12-01T10:11:41.9406974Z test_barrier_implies_wait (__main__.ProcessGroupGlooTest)
2022-12-01T10:11:41.9407404Z test_broadcast_basics (__main__.ProcessGroupGlooTest)
2022-12-01T10:11:41.9407837Z test_broadcast_basics_cuda (__main__.ProcessGroupGlooTest)
2022-12-01T10:11:41.9408285Z test_broadcast_checks (__main__.ProcessGroupGlooTest)
2022-12-01T10:11:41.9408711Z test_broadcast_stress (__main__.ProcessGroupGlooTest)
2022-12-01T10:11:41.9409123Z test_broadcast_stress_cuda (__main__.ProcessGroupGlooTest)
2022-12-01T10:11:41.9409549Z test_empty_tensors (__main__.ProcessGroupGlooTest)
2022-12-01T10:11:41.9409966Z test_gather_basics (__main__.ProcessGroupGlooTest)
2022-12-01T10:11:41.9410392Z test_gather_basics_cuda (__main__.ProcessGroupGlooTest)
2022-12-01T10:11:41.9410792Z test_gather_checks (__main__.ProcessGroupGlooTest)
2022-12-01T10:11:41.9411248Z test_gather_noncontiguous_input (__main__.ProcessGroupGlooTest)
2022-12-01T10:11:41.9411685Z test_gather_stress (__main__.ProcessGroupGlooTest)
2022-12-01T10:11:41.9412124Z test_gather_stress_cuda (__main__.ProcessGroupGlooTest)
2022-12-01T10:11:41.9412571Z test_multi_device_constructor (__main__.ProcessGroupGlooTest)
2022-12-01T10:11:41.9413030Z test_reduce_basics (__main__.ProcessGroupGlooTest)
2022-12-01T10:11:41.9413440Z test_reduce_basics_cuda (__main__.ProcessGroupGlooTest)
2022-12-01T10:11:41.9413875Z test_reduce_checks (__main__.ProcessGroupGlooTest)
2022-12-01T10:11:41.9414282Z test_reduce_stress (__main__.ProcessGroupGlooTest)
2022-12-01T10:11:41.9414695Z test_reduce_stress_cuda (__main__.ProcessGroupGlooTest)
2022-12-01T10:11:41.9415103Z test_round_robin (__main__.ProcessGroupGlooTest)
2022-12-01T10:11:41.9415545Z test_round_robin_create_destroy (__main__.ProcessGroupGlooTest)
2022-12-01T10:11:41.9415974Z test_scatter_basics (__main__.ProcessGroupGlooTest)
2022-12-01T10:11:41.9416502Z test_scatter_basics_cuda (__main__.ProcessGroupGlooTest)
2022-12-01T10:11:41.9416938Z test_scatter_checks (__main__.ProcessGroupGlooTest)
2022-12-01T10:11:41.9417412Z test_scatter_stress (__main__.ProcessGroupGlooTest)
2022-12-01T10:11:41.9417828Z test_scatter_stress_cuda (__main__.ProcessGroupGlooTest)
2022-12-01T10:11:41.9418285Z test_send_recv_all_to_all (__main__.ProcessGroupGlooTest)
2022-12-01T10:11:41.9418818Z test_sparse_allreduce_basics (__main__.ProcessGroupGlooTest)
2022-12-01T10:11:41.9419277Z test_sparse_allreduce_basics_cuda (__main__.ProcessGroupGlooTest)
2022-12-01T10:11:41.9421223Z test_sparse_allreduce_checks (__main__.ProcessGroupGlooTest)
2022-12-01T10:11:41.9422239Z <unittest.suite.TestSuite tests=[<__main__.ReducerTest testMethod=test_forward_backward>, <__main__.ReducerTest testMethod=test_forward_backward_optimizer>, <__main__.ReducerTest testMethod=test_forward_backward_unused_parameters>, <__main__.ReducerTest testMethod=test_multi_dtype_multi_bucket>, <__main__.ReducerTest testMethod=test_multi_dtype_single_bucket>, <__main__.ReducerTest testMethod=test_single_dtype_single_bucket>]>
2022-12-01T10:11:41.9423161Z test_forward_backward (__main__.ReducerTest)
2022-12-01T10:11:41.9423554Z test_forward_backward_optimizer (__main__.ReducerTest)
2022-12-01T10:11:41.9423974Z test_forward_backward_unused_parameters (__main__.ReducerTest)
2022-12-01T10:11:41.9424361Z test_multi_dtype_multi_bucket (__main__.ReducerTest)
2022-12-01T10:11:41.9424754Z test_multi_dtype_single_bucket (__main__.ReducerTest)
2022-12-01T10:11:41.9425146Z test_single_dtype_single_bucket (__main__.ReducerTest)
2022-12-01T10:11:41.9425607Z <unittest.suite.TestSuite tests=[<__main__.RendezvousEnvTest testMethod=test_logging_init>]>
2022-12-01T10:11:41.9426083Z test_logging_init (__main__.RendezvousEnvTest)
2022-12-01T10:11:41.9426445Z <unittest.suite.TestSuite tests=[]>
2022-12-01T10:11:41.9426945Z <unittest.suite.TestSuite tests=[<__main__.TimeoutTest testMethod=test_default_store_timeout_gloo>]>
2022-12-01T10:11:41.9427418Z test_default_store_timeout_gloo (__main__.TimeoutTest)
2022-12-01T10:11:41.9428178Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:11:41.9428901Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:41.9429398Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:41.9430072Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:41.9430598Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:41.9430858Z 
2022-12-01T10:11:41.9430971Z Running tests...
2022-12-01T10:11:41.9431448Z ----------------------------------------------------------------------
2022-12-01T10:11:41.9432019Z   test_broadcast_coalesced_gloo_cpu (__main__.CommTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 744
2022-12-01T10:11:41.9432677Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 745
2022-12-01T10:11:41.9433362Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:41.9433917Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:41.9434553Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:41.9435062Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:41.9435549Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:11:41.9436231Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:41.9436722Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:41.9437530Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:41.9437918Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:41.9438401Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:11:41.9438861Z ok (1.862s)
2022-12-01T10:11:41.9439051Z 
2022-12-01T10:11:41.9439440Z ----------------------------------------------------------------------
2022-12-01T10:11:41.9439748Z Ran 1 test in 1.862s
2022-12-01T10:11:41.9439865Z 
2022-12-01T10:11:41.9439927Z OK
2022-12-01T10:11:41.9440018Z 
2022-12-01T10:11:41.9440106Z Generating XML reports...
2022-12-01T10:11:41.9440512Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20221201100456.xml
2022-12-01T10:11:41.9440973Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:11:41.9441438Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:41.9441777Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:41.9442200Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:41.9442555Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:41.9442720Z 
2022-12-01T10:11:41.9442794Z Running tests...
2022-12-01T10:11:41.9443088Z ----------------------------------------------------------------------
2022-12-01T10:11:41.9443473Z   test_broadcast_coalesced_gloo_cuda (__main__.CommTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 783
2022-12-01T10:11:41.9443858Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 784
2022-12-01T10:11:41.9444309Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:41.9444630Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:41.9445063Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:41.9445416Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:41.9445741Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:11:41.9446189Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:41.9446517Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:41.9446947Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:41.9447279Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:41.9447605Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:11:41.9447897Z skip: Need at least 2 CUDA devices (1.858s)
2022-12-01T10:11:41.9448035Z 
2022-12-01T10:11:41.9448229Z ----------------------------------------------------------------------
2022-12-01T10:11:41.9448459Z Ran 1 test in 1.858s
2022-12-01T10:11:41.9448572Z 
2022-12-01T10:11:41.9448647Z OK (skipped=1)
2022-12-01T10:11:41.9448754Z 
2022-12-01T10:11:41.9448839Z Generating XML reports...
2022-12-01T10:11:41.9449231Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20221201100459.xml
2022-12-01T10:11:41.9449678Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:11:41.9450143Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:41.9450473Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:41.9450888Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:41.9451307Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:41.9451504Z 
2022-12-01T10:11:41.9451578Z Running tests...
2022-12-01T10:11:41.9451871Z ----------------------------------------------------------------------
2022-12-01T10:11:41.9452253Z   test_gloo_barrier_device_ids (__main__.CommTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 816
2022-12-01T10:11:41.9452630Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 817
2022-12-01T10:11:41.9453084Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:41.9453399Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:41.9453831Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:41.9454182Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:41.9454499Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:11:41.9454964Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:41.9455298Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:41.9455724Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:41.9456058Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:41.9456381Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:11:41.9456744Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:11:41.9457120Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:11:41.9457625Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:11:41.9458150Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:11:41.9458532Z ok (1.857s)
2022-12-01T10:11:41.9458636Z 
2022-12-01T10:11:41.9458820Z ----------------------------------------------------------------------
2022-12-01T10:11:41.9459071Z Ran 1 test in 1.857s
2022-12-01T10:11:41.9459186Z 
2022-12-01T10:11:41.9459250Z OK
2022-12-01T10:11:41.9459343Z 
2022-12-01T10:11:41.9459428Z Generating XML reports...
2022-12-01T10:11:41.9459822Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20221201100503.xml
2022-12-01T10:11:41.9460278Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:11:41.9460955Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:41.9461375Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:41.9462156Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:41.9462626Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:41.9462808Z 
2022-12-01T10:11:41.9462923Z Running tests...
2022-12-01T10:11:41.9463346Z ----------------------------------------------------------------------
2022-12-01T10:11:41.9463867Z   test_gloo_rank_membership (__main__.CommTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 855
2022-12-01T10:11:41.9464567Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 856
2022-12-01T10:11:41.9465041Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:41.9465382Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:41.9465903Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:41.9466301Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:41.9466619Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:11:41.9467079Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:41.9467413Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:41.9467851Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:41.9468190Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:41.9468522Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:11:41.9468817Z skip: Need at least 2 CUDA devices (1.860s)
2022-12-01T10:11:41.9468956Z 
2022-12-01T10:11:41.9469143Z ----------------------------------------------------------------------
2022-12-01T10:11:41.9469391Z Ran 1 test in 1.860s
2022-12-01T10:11:41.9469505Z 
2022-12-01T10:11:41.9469579Z OK (skipped=1)
2022-12-01T10:11:41.9469687Z 
2022-12-01T10:11:41.9469772Z Generating XML reports...
2022-12-01T10:11:41.9470165Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20221201100506.xml
2022-12-01T10:11:41.9470616Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:11:41.9471077Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:41.9471396Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:41.9471825Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:41.9472175Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:41.9472341Z 
2022-12-01T10:11:41.9472415Z Running tests...
2022-12-01T10:11:41.9472702Z ----------------------------------------------------------------------
2022-12-01T10:11:41.9473074Z   test_gloo_warn_not_in_group (__main__.CommTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 888
2022-12-01T10:11:41.9473443Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 889
2022-12-01T10:11:41.9473875Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:41.9474208Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:41.9474632Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:41.9474976Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:41.9475288Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:11:41.9475750Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:41.9476079Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:41.9476510Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:41.9476844Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:41.9477170Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:11:41.9477462Z skip: Need at least 2 CUDA devices (1.871s)
2022-12-01T10:11:41.9477598Z 
2022-12-01T10:11:41.9477784Z ----------------------------------------------------------------------
2022-12-01T10:11:41.9478027Z Ran 1 test in 1.871s
2022-12-01T10:11:41.9478141Z 
2022-12-01T10:11:41.9478212Z OK (skipped=1)
2022-12-01T10:11:41.9478400Z 
2022-12-01T10:11:41.9478487Z Generating XML reports...
2022-12-01T10:11:41.9478914Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20221201100510.xml
2022-12-01T10:11:41.9479369Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:11:41.9479834Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:41.9480155Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:41.9480584Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:41.9480930Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:41.9481095Z 
2022-12-01T10:11:41.9481169Z Running tests...
2022-12-01T10:11:41.9481461Z ----------------------------------------------------------------------
2022-12-01T10:11:41.9481856Z   test_sequence_num_incremented_gloo_default (__main__.CommTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 921
2022-12-01T10:11:41.9482244Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 922
2022-12-01T10:11:41.9482684Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:41.9483013Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:41.9483439Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:41.9483784Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:41.9484093Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:11:41.9484555Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:41.9484883Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:41.9485316Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:41.9485651Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:41.9485974Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:11:41.9486266Z skip: Need at least 2 CUDA devices (1.858s)
2022-12-01T10:11:41.9486403Z 
2022-12-01T10:11:41.9486586Z ----------------------------------------------------------------------
2022-12-01T10:11:41.9486837Z Ran 1 test in 1.858s
2022-12-01T10:11:41.9486950Z 
2022-12-01T10:11:41.9487008Z OK (skipped=1)
2022-12-01T10:11:41.9487119Z 
2022-12-01T10:11:41.9487203Z Generating XML reports...
2022-12-01T10:11:41.9487607Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20221201100513.xml
2022-12-01T10:11:41.9488062Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:11:41.9488513Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:41.9488844Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:41.9489272Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:41.9489618Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:41.9489768Z 
2022-12-01T10:11:41.9489840Z Running tests...
2022-12-01T10:11:41.9490140Z ----------------------------------------------------------------------
2022-12-01T10:11:41.9490530Z   test_sequence_num_incremented_gloo_subgroup (__main__.CommTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 954
2022-12-01T10:11:41.9490956Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 955
2022-12-01T10:11:41.9491407Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:41.9491765Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:41.9492195Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:41.9492527Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:41.9492848Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:11:41.9493305Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:41.9493617Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:41.9494047Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:41.9494392Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:41.9494716Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:11:41.9494995Z skip: Need at least 4 CUDA devices (1.858s)
2022-12-01T10:11:41.9495130Z 
2022-12-01T10:11:41.9495326Z ----------------------------------------------------------------------
2022-12-01T10:11:41.9495563Z Ran 1 test in 1.858s
2022-12-01T10:11:41.9495676Z 
2022-12-01T10:11:41.9495736Z OK (skipped=1)
2022-12-01T10:11:41.9495843Z 
2022-12-01T10:11:41.9495984Z Generating XML reports...
2022-12-01T10:11:41.9496477Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20221201100517.xml
2022-12-01T10:11:41.9496976Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:11:41.9497425Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:41.9497755Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:41.9498185Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:41.9498627Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:41.9498796Z 
2022-12-01T10:11:41.9498870Z Running tests...
2022-12-01T10:11:41.9499174Z ----------------------------------------------------------------------
2022-12-01T10:11:41.9499556Z   test_sequence_num_set_default_pg_gloo (__main__.CommTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 987
2022-12-01T10:11:41.9499927Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 988
2022-12-01T10:11:41.9500502Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:41.9500842Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:41.9501273Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:41.9501608Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:41.9501935Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:11:41.9502400Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:41.9502713Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:41.9503140Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:41.9503489Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:41.9503809Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:11:41.9504167Z skip: Need at least 2 CUDA devices (1.859s)
2022-12-01T10:11:41.9504310Z 
2022-12-01T10:11:41.9504550Z ----------------------------------------------------------------------
2022-12-01T10:11:41.9504794Z Ran 1 test in 1.859s
2022-12-01T10:11:41.9504906Z 
2022-12-01T10:11:41.9504967Z OK (skipped=1)
2022-12-01T10:11:41.9505074Z 
2022-12-01T10:11:41.9505161Z Generating XML reports...
2022-12-01T10:11:41.9505569Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20221201100520.xml
2022-12-01T10:11:41.9506021Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:11:41.9506469Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:41.9506843Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:41.9507340Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:41.9507820Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:41.9507993Z 
2022-12-01T10:11:41.9508099Z Running tests...
2022-12-01T10:11:41.9508473Z ----------------------------------------------------------------------
2022-12-01T10:11:41.9508915Z   test_sequence_num_set_gloo_new_group (__main__.CommTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1020
2022-12-01T10:11:41.9509320Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1021
2022-12-01T10:11:41.9509862Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:41.9510280Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:41.9510768Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:41.9511136Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:41.9511520Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:11:41.9512068Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:41.9512410Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:41.9512898Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:41.9513299Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:41.9513709Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:11:41.9514017Z skip: Need at least 2 CUDA devices (1.857s)
2022-12-01T10:11:41.9514182Z 
2022-12-01T10:11:41.9514414Z ----------------------------------------------------------------------
2022-12-01T10:11:41.9514711Z Ran 1 test in 1.858s
2022-12-01T10:11:41.9514863Z 
2022-12-01T10:11:41.9514923Z OK (skipped=1)
2022-12-01T10:11:41.9515065Z 
2022-12-01T10:11:41.9515198Z Generating XML reports...
2022-12-01T10:11:41.9515668Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20221201100524.xml
2022-12-01T10:11:41.9516180Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:11:41.9516665Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:41.9517059Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:41.9517577Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:41.9517987Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:41.9518140Z 
2022-12-01T10:11:41.9518252Z Running tests...
2022-12-01T10:11:41.9518661Z ----------------------------------------------------------------------
2022-12-01T10:11:41.9519119Z   test_tensor_dtype_complex (__main__.CommTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1053
2022-12-01T10:11:41.9519547Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1054
2022-12-01T10:11:41.9520067Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:41.9520452Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:41.9520943Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:41.9521328Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:41.9521755Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:11:41.9522279Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:41.9522625Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:41.9523122Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:41.9523553Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:41.9523938Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:11:41.9524245Z skip: Need at least 2 CUDA devices (1.859s)
2022-12-01T10:11:41.9524413Z 
2022-12-01T10:11:41.9524639Z ----------------------------------------------------------------------
2022-12-01T10:11:41.9524959Z Ran 1 test in 1.859s
2022-12-01T10:11:41.9525131Z 
2022-12-01T10:11:41.9525189Z OK (skipped=1)
2022-12-01T10:11:41.9525323Z 
2022-12-01T10:11:41.9525435Z Generating XML reports...
2022-12-01T10:11:41.9525899Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20221201100527.xml
2022-12-01T10:11:41.9526422Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:11:41.9526922Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:41.9527304Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:41.9527789Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:41.9528156Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:41.9528351Z 
2022-12-01T10:11:41.9528451Z Running tests...
2022-12-01T10:11:41.9528862Z ----------------------------------------------------------------------
2022-12-01T10:11:41.9529291Z   test_tensor_dtype_mismatch (__main__.CommTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1086
2022-12-01T10:11:41.9529693Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1087
2022-12-01T10:11:41.9530201Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:41.9530610Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:41.9531099Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:41.9531467Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:41.9531850Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:11:41.9532365Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:41.9532758Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:41.9533251Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:41.9533706Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:41.9534123Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:11:41.9534432Z skip: Need at least 2 CUDA devices (1.756s)
2022-12-01T10:11:41.9534631Z 
2022-12-01T10:11:41.9534858Z ----------------------------------------------------------------------
2022-12-01T10:11:41.9535156Z Ran 1 test in 1.757s
2022-12-01T10:11:41.9535297Z 
2022-12-01T10:11:41.9582640Z OK (skipped=1)
2022-12-01T10:11:41.9582868Z 
2022-12-01T10:11:41.9582963Z Generating XML reports...
2022-12-01T10:11:41.9583482Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20221201100531.xml
2022-12-01T10:11:41.9583929Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:11:41.9584403Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:41.9584725Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:41.9585150Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:41.9585486Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:41.9585647Z 
2022-12-01T10:11:41.9585765Z Running tests...
2022-12-01T10:11:41.9586048Z ----------------------------------------------------------------------
2022-12-01T10:11:41.9586416Z   test_allgather_work_wait_cpu (__main__.CompilerTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1119
2022-12-01T10:11:41.9586780Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1120
2022-12-01T10:11:41.9587212Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:41.9587530Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:41.9587946Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:41.9588280Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:41.9588579Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:11:41.9589026Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:41.9589339Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:41.9589745Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:41.9590071Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:41.9590378Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:11:41.9590737Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:11:41.9591088Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:11:41.9591563Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:11:41.9592077Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:11:41.9592774Z /opt/conda/lib/python3.7/site-packages/torch/fx/graph.py:1306: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-12-01T10:11:41.9593303Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-12-01T10:11:41.9594122Z /opt/conda/lib/python3.7/site-packages/torch/fx/graph.py:1306: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-12-01T10:11:41.9594707Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-12-01T10:11:41.9594947Z ok (1.858s)
2022-12-01T10:11:41.9595039Z 
2022-12-01T10:11:41.9595238Z ----------------------------------------------------------------------
2022-12-01T10:11:41.9595480Z Ran 1 test in 1.859s
2022-12-01T10:11:41.9595594Z 
2022-12-01T10:11:41.9595659Z OK
2022-12-01T10:11:41.9595752Z 
2022-12-01T10:11:41.9595825Z Generating XML reports...
2022-12-01T10:11:41.9596244Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CompilerTest-20221201100534.xml
2022-12-01T10:11:41.9596709Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:11:41.9597174Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:41.9597495Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:41.9597923Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:41.9598268Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:41.9598430Z 
2022-12-01T10:11:41.9598497Z Running tests...
2022-12-01T10:11:41.9598780Z ----------------------------------------------------------------------
2022-12-01T10:11:41.9599152Z   test_allgather_work_wait_gpu (__main__.CompilerTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1158
2022-12-01T10:11:41.9599540Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1159
2022-12-01T10:11:41.9599975Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:41.9600300Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:41.9600734Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:41.9601079Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:41.9601393Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:11:41.9601849Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:41.9602175Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:41.9602580Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:41.9602919Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:41.9603244Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:11:41.9603534Z skip: Need at least 2 CUDA devices (1.860s)
2022-12-01T10:11:41.9603660Z 
2022-12-01T10:11:41.9603857Z ----------------------------------------------------------------------
2022-12-01T10:11:41.9604093Z Ran 1 test in 1.861s
2022-12-01T10:11:41.9604207Z 
2022-12-01T10:11:41.9604280Z OK (skipped=1)
2022-12-01T10:11:41.9604387Z 
2022-12-01T10:11:41.9604458Z Generating XML reports...
2022-12-01T10:11:41.9604877Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CompilerTest-20221201100538.xml
2022-12-01T10:11:41.9605341Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:11:41.9605800Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:41.9606111Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:41.9606599Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:41.9606974Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:41.9607140Z 
2022-12-01T10:11:41.9607199Z Running tests...
2022-12-01T10:11:41.9607493Z ----------------------------------------------------------------------
2022-12-01T10:11:41.9607875Z   test_allreduce_work_wait_cpu (__main__.CompilerTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1191
2022-12-01T10:11:41.9608258Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1192
2022-12-01T10:11:41.9608696Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:41.9609023Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:41.9609451Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:41.9609794Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:41.9610099Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:11:41.9610555Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:41.9610879Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:41.9611287Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:41.9611627Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:41.9611948Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:11:41.9612312Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:11:41.9612669Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:11:41.9613160Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:11:41.9613682Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:11:41.9614376Z /opt/conda/lib/python3.7/site-packages/torch/fx/graph.py:1306: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-12-01T10:11:41.9614896Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-12-01T10:11:41.9615540Z /opt/conda/lib/python3.7/site-packages/torch/fx/graph.py:1306: UserWarning: Node _tensor_constant1 target _tensor_constant1 _tensor_constant1 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-12-01T10:11:41.9616078Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-12-01T10:11:41.9616746Z /opt/conda/lib/python3.7/site-packages/torch/fx/graph.py:1306: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-12-01T10:11:41.9617262Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-12-01T10:11:41.9617890Z /opt/conda/lib/python3.7/site-packages/torch/fx/graph.py:1306: UserWarning: Node _tensor_constant1 target _tensor_constant1 _tensor_constant1 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-12-01T10:11:41.9618505Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-12-01T10:11:41.9618748Z ok (1.857s)
2022-12-01T10:11:41.9618851Z 
2022-12-01T10:11:41.9619081Z ----------------------------------------------------------------------
2022-12-01T10:11:41.9619329Z Ran 1 test in 1.858s
2022-12-01T10:11:41.9619469Z 
2022-12-01T10:11:41.9619527Z OK
2022-12-01T10:11:41.9619612Z 
2022-12-01T10:11:41.9619694Z Generating XML reports...
2022-12-01T10:11:41.9620103Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CompilerTest-20221201100541.xml
2022-12-01T10:11:41.9620741Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:11:41.9621202Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:41.9621521Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:41.9621948Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:41.9622291Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:41.9622460Z 
2022-12-01T10:11:41.9622535Z Running tests...
2022-12-01T10:11:41.9622819Z ----------------------------------------------------------------------
2022-12-01T10:11:41.9623203Z   test_allreduce_work_wait_gpu (__main__.CompilerTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1230
2022-12-01T10:11:41.9623587Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1231
2022-12-01T10:11:41.9624026Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:41.9624348Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:41.9624770Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:41.9625119Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:41.9625432Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:11:41.9625892Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:41.9626220Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:41.9626643Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:41.9626970Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:41.9627294Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:11:41.9627578Z skip: Need at least 2 CUDA devices (1.858s)
2022-12-01T10:11:41.9627711Z 
2022-12-01T10:11:41.9627894Z ----------------------------------------------------------------------
2022-12-01T10:11:41.9628131Z Ran 1 test in 1.858s
2022-12-01T10:11:41.9628244Z 
2022-12-01T10:11:41.9628313Z OK (skipped=1)
2022-12-01T10:11:41.9628420Z 
2022-12-01T10:11:41.9628499Z Generating XML reports...
2022-12-01T10:11:41.9628903Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CompilerTest-20221201100545.xml
2022-12-01T10:11:41.9629365Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:11:41.9629828Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:41.9630140Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:41.9630560Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:41.9630903Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:41.9631068Z 
2022-12-01T10:11:41.9631137Z Running tests...
2022-12-01T10:11:41.9631423Z ----------------------------------------------------------------------
2022-12-01T10:11:41.9631865Z   test_broadcast_work_wait_cpu (__main__.CompilerTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1263
2022-12-01T10:11:41.9632250Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1264
2022-12-01T10:11:41.9632724Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:41.9633059Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:41.9633491Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:41.9633842Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:41.9634157Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:11:41.9634616Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:41.9634946Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:41.9635373Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:41.9635708Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:41.9636027Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:11:41.9636386Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:11:41.9636744Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:11:41.9637232Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:11:41.9637750Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:11:41.9638441Z /opt/conda/lib/python3.7/site-packages/torch/fx/graph.py:1306: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-12-01T10:11:41.9638968Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-12-01T10:11:41.9639610Z /opt/conda/lib/python3.7/site-packages/torch/fx/graph.py:1306: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-12-01T10:11:41.9640144Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-12-01T10:11:41.9640382Z ok (1.856s)
2022-12-01T10:11:41.9640486Z 
2022-12-01T10:11:41.9640671Z ----------------------------------------------------------------------
2022-12-01T10:11:41.9640912Z Ran 1 test in 1.856s
2022-12-01T10:11:41.9641024Z 
2022-12-01T10:11:41.9641104Z OK
2022-12-01T10:11:41.9641195Z 
2022-12-01T10:11:41.9641281Z Generating XML reports...
2022-12-01T10:11:41.9641685Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CompilerTest-20221201100548.xml
2022-12-01T10:11:41.9642151Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:11:41.9642610Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:41.9642926Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:41.9643355Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:41.9643699Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:41.9643865Z 
2022-12-01T10:11:41.9643936Z Running tests...
2022-12-01T10:11:41.9644222Z ----------------------------------------------------------------------
2022-12-01T10:11:41.9644635Z   test_broadcast_work_wait_gpu (__main__.CompilerTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1302
2022-12-01T10:11:41.9645044Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1303
2022-12-01T10:11:41.9645487Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:41.9645816Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:41.9646244Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:41.9646590Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:41.9646901Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:11:41.9647361Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:41.9647691Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:41.9648120Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:41.9648453Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:41.9648829Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:11:41.9649123Z skip: Need at least 2 CUDA devices (1.856s)
2022-12-01T10:11:41.9649260Z 
2022-12-01T10:11:41.9649442Z ----------------------------------------------------------------------
2022-12-01T10:11:41.9649683Z Ran 1 test in 1.856s
2022-12-01T10:11:41.9649796Z 
2022-12-01T10:11:41.9649867Z OK (skipped=1)
2022-12-01T10:11:41.9649974Z 
2022-12-01T10:11:41.9650059Z Generating XML reports...
2022-12-01T10:11:41.9650469Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CompilerTest-20221201100552.xml
2022-12-01T10:11:41.9650935Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:11:41.9651395Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:41.9651715Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:41.9652143Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:41.9652488Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:41.9652653Z 
2022-12-01T10:11:41.9652723Z Running tests...
2022-12-01T10:11:41.9653011Z ----------------------------------------------------------------------
2022-12-01T10:11:41.9653397Z   test_consecutive_comm_work_wait_cpu (__main__.CompilerTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1335
2022-12-01T10:11:41.9653784Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1336
2022-12-01T10:11:41.9654224Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:41.9654552Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:41.9654980Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:41.9655320Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:41.9655630Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:11:41.9656087Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:41.9656410Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:41.9656837Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:41.9657200Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:41.9657524Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:11:41.9657915Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:11:41.9658271Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:11:41.9658868Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:11:41.9659382Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:11:41.9660086Z /opt/conda/lib/python3.7/site-packages/torch/fx/graph.py:1306: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-12-01T10:11:41.9660758Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-12-01T10:11:41.9661409Z /opt/conda/lib/python3.7/site-packages/torch/fx/graph.py:1306: UserWarning: Node _tensor_constant1 target _tensor_constant1 _tensor_constant1 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-12-01T10:11:41.9661944Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-12-01T10:11:41.9662592Z /opt/conda/lib/python3.7/site-packages/torch/fx/graph.py:1306: UserWarning: Node _tensor_constant2 target _tensor_constant2 _tensor_constant2 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-12-01T10:11:41.9663126Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-12-01T10:11:41.9663760Z /opt/conda/lib/python3.7/site-packages/torch/fx/graph.py:1306: UserWarning: Node _tensor_constant3 target _tensor_constant3 _tensor_constant3 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-12-01T10:11:41.9664300Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-12-01T10:11:41.9664938Z /opt/conda/lib/python3.7/site-packages/torch/fx/graph.py:1306: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-12-01T10:11:41.9665465Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-12-01T10:11:41.9666090Z /opt/conda/lib/python3.7/site-packages/torch/fx/graph.py:1306: UserWarning: Node _tensor_constant1 target _tensor_constant1 _tensor_constant1 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-12-01T10:11:41.9666618Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-12-01T10:11:41.9667262Z /opt/conda/lib/python3.7/site-packages/torch/fx/graph.py:1306: UserWarning: Node _tensor_constant2 target _tensor_constant2 _tensor_constant2 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-12-01T10:11:41.9667791Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-12-01T10:11:41.9668429Z /opt/conda/lib/python3.7/site-packages/torch/fx/graph.py:1306: UserWarning: Node _tensor_constant3 target _tensor_constant3 _tensor_constant3 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-12-01T10:11:41.9668945Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-12-01T10:11:41.9669184Z ok (1.861s)
2022-12-01T10:11:41.9669285Z 
2022-12-01T10:11:41.9669484Z ----------------------------------------------------------------------
2022-12-01T10:11:41.9669710Z Ran 1 test in 1.861s
2022-12-01T10:11:41.9669821Z 
2022-12-01T10:11:41.9669942Z OK
2022-12-01T10:11:41.9670038Z 
2022-12-01T10:11:41.9670169Z Generating XML reports...
2022-12-01T10:11:41.9670591Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CompilerTest-20221201100555.xml
2022-12-01T10:11:41.9671043Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:11:41.9671507Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:41.9671840Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:41.9672257Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:41.9672606Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:41.9672771Z 
2022-12-01T10:11:41.9672844Z Running tests...
2022-12-01T10:11:41.9673148Z ----------------------------------------------------------------------
2022-12-01T10:11:41.9673521Z   test_consecutive_comm_work_wait_gpu (__main__.CompilerTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1374
2022-12-01T10:11:41.9673910Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1375
2022-12-01T10:11:41.9674358Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:41.9674691Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:41.9675106Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:41.9675448Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:41.9675765Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:11:41.9676213Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:41.9676540Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:41.9676967Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:41.9677307Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:41.9677614Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:11:41.9677905Z skip: Need at least 2 CUDA devices (1.757s)
2022-12-01T10:11:41.9678038Z 
2022-12-01T10:11:41.9678233Z ----------------------------------------------------------------------
2022-12-01T10:11:41.9678461Z Ran 1 test in 1.757s
2022-12-01T10:11:41.9678572Z 
2022-12-01T10:11:41.9678642Z OK (skipped=1)
2022-12-01T10:11:41.9678750Z 
2022-12-01T10:11:41.9678833Z Generating XML reports...
2022-12-01T10:11:41.9679255Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CompilerTest-20221201100559.xml
2022-12-01T10:11:41.9679748Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:11:41.9680210Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:41.9680536Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:41.9680950Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:41.9681293Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:41.9681454Z 
2022-12-01T10:11:41.9681526Z Running tests...
2022-12-01T10:11:41.9681830Z ----------------------------------------------------------------------
2022-12-01T10:11:41.9682199Z   test_nested_comm_tensor_wrapping (__main__.CompilerTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1407
2022-12-01T10:11:41.9682618Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1408
2022-12-01T10:11:41.9683101Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:41.9683430Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:41.9683845Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:41.9684189Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:41.9684515Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:11:41.9684962Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:41.9685291Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:41.9685723Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:41.9686068Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:41.9686381Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:11:41.9686746Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:11:41.9687119Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:11:41.9687601Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:11:41.9688117Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:11:41.9688822Z /opt/conda/lib/python3.7/site-packages/torch/fx/graph.py:1306: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-12-01T10:11:41.9689362Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-12-01T10:11:41.9690008Z /opt/conda/lib/python3.7/site-packages/torch/fx/graph.py:1306: UserWarning: Node _tensor_constant1 target _tensor_constant1 _tensor_constant1 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-12-01T10:11:41.9690525Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-12-01T10:11:41.9691170Z /opt/conda/lib/python3.7/site-packages/torch/fx/graph.py:1306: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-12-01T10:11:41.9691697Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-12-01T10:11:41.9692342Z /opt/conda/lib/python3.7/site-packages/torch/fx/graph.py:1306: UserWarning: Node _tensor_constant1 target _tensor_constant1 _tensor_constant1 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-12-01T10:11:41.9692865Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-12-01T10:11:41.9693104Z ok (1.854s)
2022-12-01T10:11:41.9693207Z 
2022-12-01T10:11:41.9693403Z ----------------------------------------------------------------------
2022-12-01T10:11:41.9693646Z Ran 1 test in 1.854s
2022-12-01T10:11:41.9693745Z 
2022-12-01T10:11:41.9693806Z OK
2022-12-01T10:11:41.9693896Z 
2022-12-01T10:11:41.9693980Z Generating XML reports...
2022-12-01T10:11:41.9694402Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CompilerTest-20221201100602.xml
2022-12-01T10:11:41.9694849Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:11:41.9695351Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:41.9695708Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:41.9696140Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:41.9696471Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:41.9696635Z 
2022-12-01T10:11:41.9696709Z Running tests...
2022-12-01T10:11:41.9697013Z ----------------------------------------------------------------------
2022-12-01T10:11:41.9697378Z   test_scatter_work_wait_cpu (__main__.CompilerTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1446
2022-12-01T10:11:41.9697760Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1447
2022-12-01T10:11:41.9698214Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:41.9698642Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:41.9699065Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:41.9699411Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:41.9699739Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:11:41.9700201Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:41.9700644Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:41.9701080Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:41.9701425Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:41.9701739Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:11:41.9702111Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:11:41.9702484Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:11:41.9702981Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:11:41.9703486Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:11:41.9704190Z /opt/conda/lib/python3.7/site-packages/torch/fx/graph.py:1306: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-12-01T10:11:41.9704730Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-12-01T10:11:41.9705377Z /opt/conda/lib/python3.7/site-packages/torch/fx/graph.py:1306: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-12-01T10:11:41.9705899Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-12-01T10:11:41.9706139Z ok (1.868s)
2022-12-01T10:11:41.9706248Z 
2022-12-01T10:11:41.9706443Z ----------------------------------------------------------------------
2022-12-01T10:11:41.9706684Z Ran 1 test in 1.869s
2022-12-01T10:11:41.9706784Z 
2022-12-01T10:11:41.9706846Z OK
2022-12-01T10:11:41.9706937Z 
2022-12-01T10:11:41.9707022Z Generating XML reports...
2022-12-01T10:11:41.9707441Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CompilerTest-20221201100605.xml
2022-12-01T10:11:41.9707888Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:11:41.9708403Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:41.9708773Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:41.9709201Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:41.9709533Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:41.9709699Z 
2022-12-01T10:11:41.9709773Z Running tests...
2022-12-01T10:11:41.9710075Z ----------------------------------------------------------------------
2022-12-01T10:11:41.9710442Z   test_scatter_work_wait_gpu (__main__.CompilerTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1485
2022-12-01T10:11:41.9710826Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1486
2022-12-01T10:11:41.9711279Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:41.9711611Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:41.9712028Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:41.9712370Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:41.9712689Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:11:41.9713148Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:41.9713464Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:41.9713890Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:41.9714232Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:41.9714542Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:11:41.9714833Z skip: Need at least 2 CUDA devices (1.758s)
2022-12-01T10:11:41.9714968Z 
2022-12-01T10:11:41.9715167Z ----------------------------------------------------------------------
2022-12-01T10:11:41.9715408Z Ran 1 test in 1.758s
2022-12-01T10:11:41.9715508Z 
2022-12-01T10:11:41.9715580Z OK (skipped=1)
2022-12-01T10:11:41.9715687Z 
2022-12-01T10:11:41.9715774Z Generating XML reports...
2022-12-01T10:11:41.9716195Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CompilerTest-20221201100609.xml
2022-12-01T10:11:41.9716641Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:11:41.9717101Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:41.9717430Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:41.9717862Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:41.9718198Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:41.9718363Z 
2022-12-01T10:11:41.9718435Z Running tests...
2022-12-01T10:11:41.9718736Z ----------------------------------------------------------------------
2022-12-01T10:11:41.9719051Z   test_ddp_checkpointing_dynamic_module (__main__.DistributedDataParallelTest)
2022-12-01T10:11:41.9719578Z Dynamic module can be checkpointed, multiple times, with non-reentrant ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1518
2022-12-01T10:11:41.9719985Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1519
2022-12-01T10:11:41.9720439Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:41.9720752Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:41.9721214Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:41.9721588Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:41.9721914Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:11:41.9722358Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:41.9722681Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:41.9723166Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:41.9723498Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:41.9723824Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:11:41.9724116Z skip: Need at least 2 CUDA devices (1.857s)
2022-12-01T10:11:41.9724253Z 
2022-12-01T10:11:41.9724456Z ----------------------------------------------------------------------
2022-12-01T10:11:41.9724682Z Ran 1 test in 1.857s
2022-12-01T10:11:41.9724794Z 
2022-12-01T10:11:41.9724867Z OK (skipped=1)
2022-12-01T10:11:41.9724976Z 
2022-12-01T10:11:41.9725060Z Generating XML reports...
2022-12-01T10:11:41.9725511Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221201100612.xml
2022-12-01T10:11:41.9726018Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:11:41.9726481Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:41.9726813Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:41.9727228Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:41.9727579Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:41.9727747Z 
2022-12-01T10:11:41.9727820Z Running tests...
2022-12-01T10:11:41.9728109Z ----------------------------------------------------------------------
2022-12-01T10:11:41.9728449Z   test_ddp_checkpointing_dynamic_weight_sharing (__main__.DistributedDataParallelTest)
2022-12-01T10:11:41.9728882Z Dynamic module can be checkpointed multiple times with weight sharing ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1551
2022-12-01T10:11:41.9729283Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1552
2022-12-01T10:11:41.9729721Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:41.9730053Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:41.9730483Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:41.9730830Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:41.9731142Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:11:41.9731603Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:41.9731928Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:41.9732341Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:41.9732686Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:41.9733010Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:11:41.9733301Z skip: Need at least 2 CUDA devices (1.762s)
2022-12-01T10:11:41.9733423Z 
2022-12-01T10:11:41.9733650Z ----------------------------------------------------------------------
2022-12-01T10:11:41.9733920Z Ran 1 test in 1.762s
2022-12-01T10:11:41.9734032Z 
2022-12-01T10:11:41.9734105Z OK (skipped=1)
2022-12-01T10:11:41.9734213Z 
2022-12-01T10:11:41.9734285Z Generating XML reports...
2022-12-01T10:11:41.9734751Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221201100616.xml
2022-12-01T10:11:41.9735254Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:11:41.9735716Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:41.9736033Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:41.9736459Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:41.9736809Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:41.9736977Z 
2022-12-01T10:11:41.9737036Z Running tests...
2022-12-01T10:11:41.9737341Z ----------------------------------------------------------------------
2022-12-01T10:11:41.9737675Z   test_ddp_checkpointing_once_use_reentrant_False (__main__.DistributedDataParallelTest)
2022-12-01T10:11:41.9738095Z DDP works as expected when layer is checkpointed only once. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1584
2022-12-01T10:11:41.9738580Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1585
2022-12-01T10:11:41.9739040Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:41.9739373Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:41.9739806Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:41.9740141Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:41.9740610Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:11:41.9741080Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:41.9741392Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:41.9741822Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:41.9742172Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:41.9742499Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:11:41.9742786Z skip: Need at least 2 CUDA devices (1.857s)
2022-12-01T10:11:41.9742924Z 
2022-12-01T10:11:41.9743123Z ----------------------------------------------------------------------
2022-12-01T10:11:41.9743363Z Ran 1 test in 1.857s
2022-12-01T10:11:41.9743479Z 
2022-12-01T10:11:41.9743537Z OK (skipped=1)
2022-12-01T10:11:41.9743648Z 
2022-12-01T10:11:41.9743732Z Generating XML reports...
2022-12-01T10:11:41.9744227Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221201100619.xml
2022-12-01T10:11:41.9744726Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:11:41.9745173Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:41.9745501Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:41.9745925Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:41.9746256Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:41.9746421Z 
2022-12-01T10:11:41.9746494Z Running tests...
2022-12-01T10:11:41.9746860Z ----------------------------------------------------------------------
2022-12-01T10:11:41.9747235Z   test_ddp_checkpointing_once_use_reentrant_True (__main__.DistributedDataParallelTest)
2022-12-01T10:11:41.9747641Z DDP works as expected when layer is checkpointed only once. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1617
2022-12-01T10:11:41.9748031Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1618
2022-12-01T10:11:41.9748482Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:41.9748810Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:41.9749223Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:41.9749569Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:41.9749894Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:11:41.9750339Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:41.9750668Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:41.9751093Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:41.9751434Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:41.9751739Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:11:41.9752030Z skip: Need at least 2 CUDA devices (1.756s)
2022-12-01T10:11:41.9752164Z 
2022-12-01T10:11:41.9752365Z ----------------------------------------------------------------------
2022-12-01T10:11:41.9752594Z Ran 1 test in 1.756s
2022-12-01T10:11:41.9752706Z 
2022-12-01T10:11:41.9752778Z OK (skipped=1)
2022-12-01T10:11:41.9752888Z 
2022-12-01T10:11:41.9752975Z Generating XML reports...
2022-12-01T10:11:41.9753453Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221201100623.xml
2022-12-01T10:11:41.9753936Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:11:41.9754397Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:41.9754729Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:41.9755144Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:41.9755488Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:41.9755652Z 
2022-12-01T10:11:41.9755725Z Running tests...
2022-12-01T10:11:41.9756027Z ----------------------------------------------------------------------
2022-12-01T10:11:41.9756375Z   test_ddp_checkpointing_twice_static_graph_use_reentrant_False (__main__.DistributedDataParallelTest)
2022-12-01T10:11:41.9756918Z Regardless of reentrant or non-reentrant checkpointing impl, ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1650
2022-12-01T10:11:41.9757316Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1651
2022-12-01T10:11:41.9757765Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:41.9758083Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:41.9758513Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:41.9758856Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:41.9759165Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:11:41.9759656Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:41.9760010Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:41.9760441Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:41.9760773Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:41.9761092Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:11:41.9761382Z skip: Need at least 2 CUDA devices (1.855s)
2022-12-01T10:11:41.9761518Z 
2022-12-01T10:11:41.9761702Z ----------------------------------------------------------------------
2022-12-01T10:11:41.9761943Z Ran 1 test in 1.856s
2022-12-01T10:11:41.9762056Z 
2022-12-01T10:11:41.9762130Z OK (skipped=1)
2022-12-01T10:11:41.9762238Z 
2022-12-01T10:11:41.9762323Z Generating XML reports...
2022-12-01T10:11:41.9762781Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221201100626.xml
2022-12-01T10:11:41.9763289Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:11:41.9763753Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:41.9764070Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:41.9764497Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:41.9764844Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:41.9765009Z 
2022-12-01T10:11:41.9765082Z Running tests...
2022-12-01T10:11:41.9765368Z ----------------------------------------------------------------------
2022-12-01T10:11:41.9765723Z   test_ddp_checkpointing_twice_static_graph_use_reentrant_True (__main__.DistributedDataParallelTest)
2022-12-01T10:11:41.9766262Z Regardless of reentrant or non-reentrant checkpointing impl, ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1683
2022-12-01T10:11:41.9766665Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1684
2022-12-01T10:11:41.9767102Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:41.9767430Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:41.9767857Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:41.9768189Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:41.9768512Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:11:41.9768970Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:41.9769298Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:41.9769713Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:41.9770059Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:41.9770381Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:11:41.9770655Z skip: Need at least 2 CUDA devices (1.856s)
2022-12-01T10:11:41.9770791Z 
2022-12-01T10:11:41.9770985Z ----------------------------------------------------------------------
2022-12-01T10:11:41.9771226Z Ran 1 test in 1.857s
2022-12-01T10:11:41.9771338Z 
2022-12-01T10:11:41.9771410Z OK (skipped=1)
2022-12-01T10:11:41.9771505Z 
2022-12-01T10:11:41.9771588Z Generating XML reports...
2022-12-01T10:11:41.9772086Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221201100629.xml
2022-12-01T10:11:41.9772589Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:11:41.9773066Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:41.9773396Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:41.9773823Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:41.9774174Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:41.9774338Z 
2022-12-01T10:11:41.9774396Z Running tests...
2022-12-01T10:11:41.9774699Z ----------------------------------------------------------------------
2022-12-01T10:11:41.9775044Z   test_ddp_checkpointing_twice_use_reentrant_False (__main__.DistributedDataParallelTest)
2022-12-01T10:11:41.9775589Z Checkpoitning twice fails for non-static graph with reentrant checkpoint ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1716
2022-12-01T10:11:41.9775987Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1717
2022-12-01T10:11:41.9776437Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:41.9776768Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:41.9777183Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:41.9777530Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:41.9777856Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:11:41.9778315Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:41.9778702Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:41.9779136Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:41.9779487Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:41.9779794Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:11:41.9780083Z skip: Need at least 2 CUDA devices (1.759s)
2022-12-01T10:11:41.9780220Z 
2022-12-01T10:11:41.9780540Z ----------------------------------------------------------------------
2022-12-01T10:11:41.9780786Z Ran 1 test in 1.759s
2022-12-01T10:11:41.9780885Z 
2022-12-01T10:11:41.9780958Z OK (skipped=1)
2022-12-01T10:11:41.9781067Z 
2022-12-01T10:11:41.9781152Z Generating XML reports...
2022-12-01T10:11:41.9781618Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221201100633.xml
2022-12-01T10:11:41.9782109Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:11:41.9782573Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:41.9782909Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:41.9783339Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:41.9783672Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:41.9783839Z 
2022-12-01T10:11:41.9783911Z Running tests...
2022-12-01T10:11:41.9784209Z ----------------------------------------------------------------------
2022-12-01T10:11:41.9784536Z   test_ddp_checkpointing_twice_use_reentrant_True (__main__.DistributedDataParallelTest)
2022-12-01T10:11:41.9785080Z Checkpoitning twice fails for non-static graph with reentrant checkpoint ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1749
2022-12-01T10:11:41.9785560Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1750
2022-12-01T10:11:41.9786047Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:41.9786365Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:41.9786793Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:41.9787140Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:41.9787463Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:11:41.9787905Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:41.9788232Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:41.9788661Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:41.9788992Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:41.9789315Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:11:41.9789608Z skip: Need at least 2 CUDA devices (1.863s)
2022-12-01T10:11:41.9789741Z 
2022-12-01T10:11:41.9789938Z ----------------------------------------------------------------------
2022-12-01T10:11:41.9790165Z Ran 1 test in 1.863s
2022-12-01T10:11:41.9790278Z 
2022-12-01T10:11:41.9790352Z OK (skipped=1)
2022-12-01T10:11:41.9790459Z 
2022-12-01T10:11:41.9790542Z Generating XML reports...
2022-12-01T10:11:41.9790993Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221201100636.xml
2022-12-01T10:11:41.9791495Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:11:41.9791965Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:41.9792298Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:41.9792713Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:41.9793059Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:41.9793225Z 
2022-12-01T10:11:41.9793297Z Running tests...
2022-12-01T10:11:41.9793598Z ----------------------------------------------------------------------
2022-12-01T10:11:41.9793924Z   test_ddp_checkpointing_twice_weight_sharing (__main__.DistributedDataParallelTest)
2022-12-01T10:11:41.9794354Z Checkpointing should work with static graph in the case of checkpointing ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1782
2022-12-01T10:11:41.9794765Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1783
2022-12-01T10:11:41.9795206Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:41.9795538Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:41.9795970Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:41.9796313Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:41.9796626Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:11:41.9797084Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:41.9797411Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:41.9797823Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:41.9798226Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:41.9798618Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:11:41.9798907Z skip: Need at least 2 CUDA devices (1.856s)
2022-12-01T10:11:41.9799045Z 
2022-12-01T10:11:41.9799231Z ----------------------------------------------------------------------
2022-12-01T10:11:41.9799471Z Ran 1 test in 1.856s
2022-12-01T10:11:41.9799585Z 
2022-12-01T10:11:41.9799658Z OK (skipped=1)
2022-12-01T10:11:41.9799766Z 
2022-12-01T10:11:41.9799837Z Generating XML reports...
2022-12-01T10:11:41.9800306Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221201100640.xml
2022-12-01T10:11:41.9800808Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:11:41.9801271Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:41.9801589Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:41.9802021Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:41.9802365Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:41.9802528Z 
2022-12-01T10:11:41.9802600Z Running tests...
2022-12-01T10:11:41.9802888Z ----------------------------------------------------------------------
2022-12-01T10:11:41.9803242Z   test_ddp_checkpointing_unused_params_use_reentrant_False (__main__.DistributedDataParallelTest)
2022-12-01T10:11:41.9803684Z With reentrant autograd checkpointing impl, DDP will fail when there are ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1815
2022-12-01T10:11:41.9804076Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1816
2022-12-01T10:11:41.9804532Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:41.9804861Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:41.9805291Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:41.9805623Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:41.9805944Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:11:41.9806403Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:41.9806716Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:41.9807142Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:41.9807483Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:41.9807808Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:11:41.9808084Z skip: Need at least 2 CUDA devices (1.758s)
2022-12-01T10:11:41.9808223Z 
2022-12-01T10:11:41.9808419Z ----------------------------------------------------------------------
2022-12-01T10:11:41.9808661Z Ran 1 test in 1.758s
2022-12-01T10:11:41.9808773Z 
2022-12-01T10:11:41.9808878Z OK (skipped=1)
2022-12-01T10:11:41.9808972Z 
2022-12-01T10:11:41.9809055Z Generating XML reports...
2022-12-01T10:11:41.9809519Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221201100643.xml
2022-12-01T10:11:41.9810019Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:11:41.9810463Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:41.9810793Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:41.9811249Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:41.9811628Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:41.9811777Z 
2022-12-01T10:11:41.9811850Z Running tests...
2022-12-01T10:11:41.9812150Z ----------------------------------------------------------------------
2022-12-01T10:11:41.9812500Z   test_ddp_checkpointing_unused_params_use_reentrant_True (__main__.DistributedDataParallelTest)
2022-12-01T10:11:41.9812923Z With reentrant autograd checkpointing impl, DDP will fail when there are ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1848
2022-12-01T10:11:41.9813325Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1849
2022-12-01T10:11:41.9813775Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:41.9814107Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:41.9814523Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:41.9814660Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:41.9814824Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:11:41.9815092Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:41.9815214Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:41.9815490Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:41.9815624Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:41.9815788Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:11:41.9815880Z skip: Need at least 2 CUDA devices (1.860s)
2022-12-01T10:11:41.9815887Z 
2022-12-01T10:11:41.9816079Z ----------------------------------------------------------------------
2022-12-01T10:11:41.9816154Z Ran 1 test in 1.860s
2022-12-01T10:11:41.9816159Z 
2022-12-01T10:11:41.9816232Z OK (skipped=1)
2022-12-01T10:11:41.9816237Z 
2022-12-01T10:11:41.9816320Z Generating XML reports...
2022-12-01T10:11:41.9816670Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221201100646.xml
2022-12-01T10:11:41.9816989Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:11:41.9817263Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:41.9817372Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:41.9817655Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:41.9817793Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:41.9817798Z 
2022-12-01T10:11:41.9817872Z Running tests...
2022-12-01T10:11:41.9818068Z ----------------------------------------------------------------------
2022-12-01T10:11:41.9818257Z   test_ddp_checkpointing_weight_sharing_use_reentrant_False (__main__.DistributedDataParallelTest)
2022-12-01T10:11:41.9818557Z Test that checkpointing with weight sharing works. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1881
2022-12-01T10:11:41.9818716Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1882
2022-12-01T10:11:41.9818993Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:41.9819103Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:41.9819427Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:41.9819591Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:41.9819754Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:11:41.9820025Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:41.9820148Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:41.9820530Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:41.9820666Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:41.9820815Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:11:41.9820920Z skip: Need at least 2 CUDA devices (1.757s)
2022-12-01T10:11:41.9820925Z 
2022-12-01T10:11:41.9821123Z ----------------------------------------------------------------------
2022-12-01T10:11:41.9821202Z Ran 1 test in 1.757s
2022-12-01T10:11:41.9821207Z 
2022-12-01T10:11:41.9821279Z OK (skipped=1)
2022-12-01T10:11:41.9821284Z 
2022-12-01T10:11:41.9821373Z Generating XML reports...
2022-12-01T10:11:41.9821728Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221201100650.xml
2022-12-01T10:11:41.9821960Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:11:41.9822233Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:41.9822343Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:41.9822625Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:41.9822763Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:41.9822769Z 
2022-12-01T10:11:41.9822846Z Running tests...
2022-12-01T10:11:41.9823045Z ----------------------------------------------------------------------
2022-12-01T10:11:41.9823231Z   test_ddp_checkpointing_weight_sharing_use_reentrant_True (__main__.DistributedDataParallelTest)
2022-12-01T10:11:41.9823444Z Test that checkpointing with weight sharing works. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1914
2022-12-01T10:11:41.9823599Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1915
2022-12-01T10:11:41.9823872Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:41.9823983Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:41.9824265Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:41.9824403Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:41.9824569Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:11:41.9824840Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:41.9824961Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:41.9825237Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:41.9825369Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:41.9825516Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:11:41.9825621Z skip: Need at least 2 CUDA devices (1.857s)
2022-12-01T10:11:41.9825626Z 
2022-12-01T10:11:41.9825821Z ----------------------------------------------------------------------
2022-12-01T10:11:41.9825949Z Ran 1 test in 1.857s
2022-12-01T10:11:41.9825956Z 
2022-12-01T10:11:41.9826063Z OK (skipped=1)
2022-12-01T10:11:41.9826068Z 
2022-12-01T10:11:41.9826154Z Generating XML reports...
2022-12-01T10:11:41.9826501Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221201100653.xml
2022-12-01T10:11:41.9826734Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:11:41.9827007Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:41.9827117Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:41.9827398Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:41.9827532Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:41.9827538Z 
2022-12-01T10:11:41.9827613Z Running tests...
2022-12-01T10:11:41.9827810Z ----------------------------------------------------------------------
2022-12-01T10:11:41.9827971Z   test_ddp_comm_hook_future_passing_cpu (__main__.DistributedDataParallelTest)
2022-12-01T10:11:41.9828208Z This unit test verifies whether the Future object is passed properly. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1947
2022-12-01T10:11:41.9828362Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1948
2022-12-01T10:11:41.9828622Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:41.9828746Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:41.9829026Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:41.9829163Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:41.9829332Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:11:41.9829606Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:41.9829729Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:41.9830004Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:41.9830137Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:41.9830287Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:11:41.9830472Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpufnq4vyg
2022-12-01T10:11:41.9830660Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppo9p2lh1
2022-12-01T10:11:41.9830860Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpufnq4vyg/_remote_module_non_scriptable.py
2022-12-01T10:11:41.9831054Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppo9p2lh1/_remote_module_non_scriptable.py
2022-12-01T10:11:41.9831129Z ok (1.859s)
2022-12-01T10:11:41.9831134Z 
2022-12-01T10:11:41.9831328Z ----------------------------------------------------------------------
2022-12-01T10:11:41.9831405Z Ran 1 test in 1.859s
2022-12-01T10:11:41.9831410Z 
2022-12-01T10:11:41.9831457Z OK
2022-12-01T10:11:41.9831476Z 
2022-12-01T10:11:41.9831546Z Generating XML reports...
2022-12-01T10:11:41.9831895Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221201100657.xml
2022-12-01T10:11:41.9832127Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:11:41.9832400Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:41.9832556Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:41.9832838Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:41.9833002Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:41.9833008Z 
2022-12-01T10:11:41.9833080Z Running tests...
2022-12-01T10:11:41.9833261Z ----------------------------------------------------------------------
2022-12-01T10:11:41.9833428Z   test_ddp_comm_hook_future_passing_gpu_gloo (__main__.DistributedDataParallelTest)
2022-12-01T10:11:41.9833681Z This unit test verifies whether the Future object is passed properly using gloo backend. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1986
2022-12-01T10:11:41.9833838Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1987
2022-12-01T10:11:41.9834109Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:41.9834235Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:41.9834518Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:41.9834652Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:41.9834816Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:11:41.9835070Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:41.9835193Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:41.9835470Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:41.9835604Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:41.9835768Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:11:41.9835873Z skip: Need at least 2 CUDA devices (1.858s)
2022-12-01T10:11:41.9835880Z 
2022-12-01T10:11:41.9836071Z ----------------------------------------------------------------------
2022-12-01T10:11:41.9836149Z Ran 1 test in 1.858s
2022-12-01T10:11:41.9836154Z 
2022-12-01T10:11:41.9836212Z OK (skipped=1)
2022-12-01T10:11:41.9836217Z 
2022-12-01T10:11:41.9836303Z Generating XML reports...
2022-12-01T10:11:41.9836647Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221201100700.xml
2022-12-01T10:11:41.9836881Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:11:41.9837153Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:41.9837275Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:41.9837558Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:41.9837695Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:41.9837700Z 
2022-12-01T10:11:41.9837772Z Running tests...
2022-12-01T10:11:41.9837952Z ----------------------------------------------------------------------
2022-12-01T10:11:41.9838109Z   test_ddp_comm_hook_register_just_once (__main__.DistributedDataParallelTest)
2022-12-01T10:11:41.9838356Z DDP communication hook can only be registered once. This test validates whether ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2019
2022-12-01T10:11:41.9838512Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2020
2022-12-01T10:11:41.9838785Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:41.9838909Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:41.9839216Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:41.9839381Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:41.9839531Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:11:41.9839802Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:41.9839924Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:41.9840200Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:41.9840333Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:41.9840495Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:11:41.9840683Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6xdbja3e
2022-12-01T10:11:41.9840873Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp15ob3o_z
2022-12-01T10:11:41.9841071Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6xdbja3e/_remote_module_non_scriptable.py
2022-12-01T10:11:41.9841252Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp15ob3o_z/_remote_module_non_scriptable.py
2022-12-01T10:11:41.9841321Z ok (1.857s)
2022-12-01T10:11:41.9841327Z 
2022-12-01T10:11:41.9841525Z ----------------------------------------------------------------------
2022-12-01T10:11:41.9841601Z Ran 1 test in 1.857s
2022-12-01T10:11:41.9841606Z 
2022-12-01T10:11:41.9841666Z OK
2022-12-01T10:11:41.9841671Z 
2022-12-01T10:11:41.9841755Z Generating XML reports...
2022-12-01T10:11:41.9842101Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221201100704.xml
2022-12-01T10:11:41.9842336Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:11:41.9842599Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:41.9842724Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:41.9843003Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:41.9843139Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:41.9843144Z 
2022-12-01T10:11:41.9843216Z Running tests...
2022-12-01T10:11:41.9843411Z ----------------------------------------------------------------------
2022-12-01T10:11:41.9843572Z   test_ddp_comm_hook_sparse_gradients (__main__.DistributedDataParallelTest)
2022-12-01T10:11:41.9843812Z Runs "test_sparse_gradients" unit test with DDP communication hook. We define a ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2058
2022-12-01T10:11:41.9843969Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2059
2022-12-01T10:11:41.9844231Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:41.9844353Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:41.9844632Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:41.9844767Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:41.9844933Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:11:41.9845202Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:41.9845327Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:41.9845649Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:41.9845818Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:41.9845968Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:11:41.9846155Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk_9zs54i
2022-12-01T10:11:41.9846340Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1eoqaa51
2022-12-01T10:11:41.9846533Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk_9zs54i/_remote_module_non_scriptable.py
2022-12-01T10:11:41.9846730Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1eoqaa51/_remote_module_non_scriptable.py
2022-12-01T10:11:41.9846799Z ok (1.859s)
2022-12-01T10:11:41.9846804Z 
2022-12-01T10:11:41.9847002Z ----------------------------------------------------------------------
2022-12-01T10:11:41.9847077Z Ran 1 test in 1.859s
2022-12-01T10:11:41.9847085Z 
2022-12-01T10:11:41.9847133Z OK
2022-12-01T10:11:41.9847140Z 
2022-12-01T10:11:41.9847224Z Generating XML reports...
2022-12-01T10:11:41.9847576Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221201100707.xml
2022-12-01T10:11:41.9847808Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:11:41.9848080Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:41.9848204Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:41.9848483Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:41.9848619Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:41.9848625Z 
2022-12-01T10:11:41.9848697Z Running tests...
2022-12-01T10:11:41.9848881Z ----------------------------------------------------------------------
2022-12-01T10:11:41.9849035Z   test_ddp_invalid_comm_hook_init (__main__.DistributedDataParallelTest)
2022-12-01T10:11:41.9849272Z This unit test makes sure that register_comm_hook properly checks the format ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2103
2022-12-01T10:11:41.9849429Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2104
2022-12-01T10:11:41.9849699Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:41.9849823Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:41.9850105Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:41.9850239Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:41.9850393Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:11:41.9850664Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:41.9850787Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:41.9851063Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:41.9851196Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:41.9851360Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:11:41.9851545Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpscrgkgea
2022-12-01T10:11:41.9851727Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp47sfe_30
2022-12-01T10:11:41.9851923Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpscrgkgea/_remote_module_non_scriptable.py
2022-12-01T10:11:41.9852127Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp47sfe_30/_remote_module_non_scriptable.py
2022-12-01T10:11:41.9852223Z ok (1.861s)
2022-12-01T10:11:41.9852228Z 
2022-12-01T10:11:41.9852423Z ----------------------------------------------------------------------
2022-12-01T10:11:41.9852498Z Ran 1 test in 1.861s
2022-12-01T10:11:41.9852503Z 
2022-12-01T10:11:41.9852565Z OK
2022-12-01T10:11:41.9852570Z 
2022-12-01T10:11:41.9852657Z Generating XML reports...
2022-12-01T10:11:41.9853008Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221201100711.xml
2022-12-01T10:11:41.9853241Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:11:41.9853500Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:41.9853626Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:41.9853909Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:41.9854048Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:41.9854053Z 
2022-12-01T10:11:41.9854126Z Running tests...
2022-12-01T10:11:41.9854323Z ----------------------------------------------------------------------
2022-12-01T10:11:41.9854487Z   test_ddp_invalid_comm_hook_return_type (__main__.DistributedDataParallelTest)
2022-12-01T10:11:41.9854737Z This test checks whether return annotation checked properly if defined. It also ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2142
2022-12-01T10:11:41.9854893Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2143
2022-12-01T10:11:41.9855155Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:41.9855282Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:41.9855566Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:41.9855706Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:41.9855872Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:11:41.9856143Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:41.9856268Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:41.9856549Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:41.9856670Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:41.9856833Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:11:41.9857023Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk3bjsr__
2022-12-01T10:11:41.9857210Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpns60j4xc
2022-12-01T10:11:41.9857406Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk3bjsr__/_remote_module_non_scriptable.py
2022-12-01T10:11:41.9857597Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpns60j4xc/_remote_module_non_scriptable.py
2022-12-01T10:11:41.9857666Z ok (1.858s)
2022-12-01T10:11:41.9857671Z 
2022-12-01T10:11:41.9857870Z ----------------------------------------------------------------------
2022-12-01T10:11:41.9857945Z Ran 1 test in 1.858s
2022-12-01T10:11:41.9857951Z 
2022-12-01T10:11:41.9857999Z OK
2022-12-01T10:11:41.9858004Z 
2022-12-01T10:11:41.9858088Z Generating XML reports...
2022-12-01T10:11:41.9858525Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221201100714.xml
2022-12-01T10:11:41.9858807Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:11:41.9859113Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:41.9859237Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:41.9859520Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:41.9859657Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:41.9859664Z 
2022-12-01T10:11:41.9859724Z Running tests...
2022-12-01T10:11:41.9859922Z ----------------------------------------------------------------------
2022-12-01T10:11:41.9860108Z   test_find_unused_parameters_when_unused_parameters_empty (__main__.DistributedDataParallelTest)
2022-12-01T10:11:41.9860441Z An empty unused_parameters array does not imply find_unused_parameters = ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2181
2022-12-01T10:11:41.9860603Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2182
2022-12-01T10:11:41.9860879Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:41.9861003Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:41.9861288Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:41.9861425Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:41.9861578Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:11:41.9861849Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:41.9861972Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:41.9862253Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:41.9862390Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:41.9862552Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:11:41.9862659Z skip: Need at least 2 CUDA devices (1.860s)
2022-12-01T10:11:41.9862665Z 
2022-12-01T10:11:41.9862861Z ----------------------------------------------------------------------
2022-12-01T10:11:41.9862924Z Ran 1 test in 1.860s
2022-12-01T10:11:41.9862945Z 
2022-12-01T10:11:41.9863006Z OK (skipped=1)
2022-12-01T10:11:41.9863011Z 
2022-12-01T10:11:41.9863097Z Generating XML reports...
2022-12-01T10:11:41.9863443Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221201100717.xml
2022-12-01T10:11:41.9863676Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:11:41.9863949Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:41.9864074Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:41.9864353Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:41.9864489Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:41.9864494Z 
2022-12-01T10:11:41.9864552Z Running tests...
2022-12-01T10:11:41.9864748Z ----------------------------------------------------------------------
2022-12-01T10:11:41.9864998Z   test_global_local_unused_params_grad (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2214
2022-12-01T10:11:41.9865155Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2215
2022-12-01T10:11:41.9865477Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:41.9865636Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:41.9865918Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:41.9866056Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:41.9866221Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:11:41.9866478Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:41.9866603Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:41.9866880Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:41.9867015Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:41.9867179Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:11:41.9867288Z skip: Need at least 2 CUDA devices (1.855s)
2022-12-01T10:11:41.9867294Z 
2022-12-01T10:11:41.9867488Z ----------------------------------------------------------------------
2022-12-01T10:11:41.9867564Z Ran 1 test in 1.855s
2022-12-01T10:11:41.9867570Z 
2022-12-01T10:11:41.9867627Z OK (skipped=1)
2022-12-01T10:11:41.9867647Z 
2022-12-01T10:11:41.9867718Z Generating XML reports...
2022-12-01T10:11:41.9868067Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221201100721.xml
2022-12-01T10:11:41.9868298Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:11:41.9868569Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:41.9868695Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:41.9868976Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:41.9869109Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:41.9869114Z 
2022-12-01T10:11:41.9869186Z Running tests...
2022-12-01T10:11:41.9869367Z ----------------------------------------------------------------------
2022-12-01T10:11:41.9869635Z   test_global_local_unused_params_grad_with_grad_is_view (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2247
2022-12-01T10:11:41.9869791Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2248
2022-12-01T10:11:41.9870065Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:41.9870188Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:41.9870469Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:41.9870605Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:41.9870769Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:11:41.9871045Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:41.9871153Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:41.9871434Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:41.9871569Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:41.9871731Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:11:41.9871869Z skip: Need at least 2 CUDA devices (1.860s)
2022-12-01T10:11:41.9871876Z 
2022-12-01T10:11:41.9872101Z ----------------------------------------------------------------------
2022-12-01T10:11:41.9872176Z Ran 1 test in 1.860s
2022-12-01T10:11:41.9872182Z 
2022-12-01T10:11:41.9872252Z OK (skipped=1)
2022-12-01T10:11:41.9872257Z 
2022-12-01T10:11:41.9872326Z Generating XML reports...
2022-12-01T10:11:41.9872674Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221201100724.xml
2022-12-01T10:11:41.9872906Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:11:41.9873181Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:41.9873304Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:41.9873586Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:41.9873726Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:41.9873733Z 
2022-12-01T10:11:41.9873806Z Running tests...
2022-12-01T10:11:41.9874000Z ----------------------------------------------------------------------
2022-12-01T10:11:41.9874252Z   test_global_local_unused_params_grad_with_static_graph (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2280
2022-12-01T10:11:41.9874407Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2281
2022-12-01T10:11:41.9874679Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:41.9874803Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:41.9875082Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:41.9875218Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:41.9875385Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:11:41.9875652Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:41.9875773Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:41.9876035Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:41.9876168Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:41.9876330Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:11:41.9876438Z skip: Need at least 2 CUDA devices (1.859s)
2022-12-01T10:11:41.9876444Z 
2022-12-01T10:11:41.9876637Z ----------------------------------------------------------------------
2022-12-01T10:11:41.9876714Z Ran 1 test in 1.859s
2022-12-01T10:11:41.9876719Z 
2022-12-01T10:11:41.9876792Z OK (skipped=1)
2022-12-01T10:11:41.9876797Z 
2022-12-01T10:11:41.9876880Z Generating XML reports...
2022-12-01T10:11:41.9877213Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221201100728.xml
2022-12-01T10:11:41.9877443Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:11:41.9877715Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:41.9877838Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:41.9878117Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:41.9878252Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:41.9878257Z 
2022-12-01T10:11:41.9878329Z Running tests...
2022-12-01T10:11:41.9878600Z ----------------------------------------------------------------------
2022-12-01T10:11:41.9878897Z   test_gloo_backend_1gpu_module_device_ids_integer_list (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2313
2022-12-01T10:11:41.9879039Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2314
2022-12-01T10:11:41.9879310Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:41.9879432Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:41.9879710Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:41.9879846Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:41.9880009Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:11:41.9880281Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:41.9880404Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:41.9880679Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:41.9880799Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:41.9880960Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:11:41.9881065Z skip: Need at least 2 CUDA devices (1.855s)
2022-12-01T10:11:41.9881070Z 
2022-12-01T10:11:41.9881264Z ----------------------------------------------------------------------
2022-12-01T10:11:41.9881340Z Ran 1 test in 1.855s
2022-12-01T10:11:41.9881345Z 
2022-12-01T10:11:41.9881415Z OK (skipped=1)
2022-12-01T10:11:41.9881420Z 
2022-12-01T10:11:41.9881505Z Generating XML reports...
2022-12-01T10:11:41.9881855Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221201100731.xml
2022-12-01T10:11:41.9882076Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:11:41.9882349Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:41.9882472Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:41.9882749Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:41.9882886Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:41.9882892Z 
2022-12-01T10:11:41.9882965Z Running tests...
2022-12-01T10:11:41.9883161Z ----------------------------------------------------------------------
2022-12-01T10:11:41.9883435Z   test_gloo_backend_1gpu_module_device_ids_torch_device_list (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2346
2022-12-01T10:11:41.9883594Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2347
2022-12-01T10:11:41.9883854Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:41.9883977Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:41.9884257Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:41.9884393Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:41.9884562Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:11:41.9884835Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:41.9884992Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:41.9885270Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:41.9885433Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:41.9885582Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:11:41.9885687Z skip: Need at least 2 CUDA devices (1.855s)
2022-12-01T10:11:41.9885692Z 
2022-12-01T10:11:41.9885892Z ----------------------------------------------------------------------
2022-12-01T10:11:41.9885966Z Ran 1 test in 1.855s
2022-12-01T10:11:41.9885971Z 
2022-12-01T10:11:41.9886046Z OK (skipped=1)
2022-12-01T10:11:41.9886051Z 
2022-12-01T10:11:41.9886137Z Generating XML reports...
2022-12-01T10:11:41.9886485Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221201100735.xml
2022-12-01T10:11:41.9886720Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:11:41.9886983Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:41.9887107Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:41.9887387Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:41.9887521Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:41.9887526Z 
2022-12-01T10:11:41.9887599Z Running tests...
2022-12-01T10:11:41.9887796Z ----------------------------------------------------------------------
2022-12-01T10:11:41.9888040Z   test_gloo_backend_2gpu_module (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2379
2022-12-01T10:11:41.9888198Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2380
2022-12-01T10:11:41.9888473Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:41.9888585Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:41.9888862Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:41.9888995Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:41.9889160Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:11:41.9889428Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:41.9889549Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:41.9889828Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:41.9889965Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:41.9890114Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:11:41.9890227Z skip: Need at least 4 CUDA devices (1.858s)
2022-12-01T10:11:41.9890232Z 
2022-12-01T10:11:41.9890426Z ----------------------------------------------------------------------
2022-12-01T10:11:41.9890502Z Ran 1 test in 1.858s
2022-12-01T10:11:41.9890507Z 
2022-12-01T10:11:41.9890578Z OK (skipped=1)
2022-12-01T10:11:41.9890583Z 
2022-12-01T10:11:41.9890667Z Generating XML reports...
2022-12-01T10:11:41.9891014Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221201100738.xml
2022-12-01T10:11:41.9891242Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:11:41.9891514Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:41.9891664Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:41.9891971Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:41.9892105Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:41.9892111Z 
2022-12-01T10:11:41.9892184Z Running tests...
2022-12-01T10:11:41.9892380Z ----------------------------------------------------------------------
2022-12-01T10:11:41.9892627Z   test_gloo_backend_4gpu_module (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2412
2022-12-01T10:11:41.9892782Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2413
2022-12-01T10:11:41.9893052Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:41.9893178Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:41.9893443Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:41.9893581Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:41.9893744Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:11:41.9894012Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:41.9894133Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:41.9894408Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:41.9894541Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:41.9894706Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:11:41.9894802Z skip: Need at least 8 CUDA devices (1.859s)
2022-12-01T10:11:41.9894810Z 
2022-12-01T10:11:41.9895007Z ----------------------------------------------------------------------
2022-12-01T10:11:41.9895084Z Ran 1 test in 1.860s
2022-12-01T10:11:41.9895089Z 
2022-12-01T10:11:41.9895160Z OK (skipped=1)
2022-12-01T10:11:41.9895165Z 
2022-12-01T10:11:41.9895248Z Generating XML reports...
2022-12-01T10:11:41.9895594Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221201100742.xml
2022-12-01T10:11:41.9895825Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:11:41.9896095Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:41.9896218Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:41.9896485Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:41.9896619Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:41.9896627Z 
2022-12-01T10:11:41.9896700Z Running tests...
2022-12-01T10:11:41.9896897Z ----------------------------------------------------------------------
2022-12-01T10:11:41.9897139Z   test_gloo_backend_cpu_module (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2445
2022-12-01T10:11:41.9897294Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2446
2022-12-01T10:11:41.9897567Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:41.9897689Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:41.9897958Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:41.9898125Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:41.9898321Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:11:41.9898692Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:41.9898816Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:41.9899094Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:41.9899227Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:41.9899393Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:11:41.9899580Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5u664yqk
2022-12-01T10:11:41.9899750Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5d_1y4cn
2022-12-01T10:11:41.9899951Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5u664yqk/_remote_module_non_scriptable.py
2022-12-01T10:11:41.9900145Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5d_1y4cn/_remote_module_non_scriptable.py
2022-12-01T10:11:41.9900321Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:11:41.9900605Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:11:41.9900675Z ok (1.855s)
2022-12-01T10:11:41.9900680Z 
2022-12-01T10:11:41.9900878Z ----------------------------------------------------------------------
2022-12-01T10:11:41.9900954Z Ran 1 test in 1.855s
2022-12-01T10:11:41.9900959Z 
2022-12-01T10:11:41.9901006Z OK
2022-12-01T10:11:41.9901025Z 
2022-12-01T10:11:41.9901095Z Generating XML reports...
2022-12-01T10:11:41.9901443Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221201100745.xml
2022-12-01T10:11:41.9901676Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:11:41.9901954Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:41.9902079Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:41.9902358Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:41.9902494Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:41.9902498Z 
2022-12-01T10:11:41.9902571Z Running tests...
2022-12-01T10:11:41.9902749Z ----------------------------------------------------------------------
2022-12-01T10:11:41.9903007Z   test_gloo_backend_cpu_module_grad_is_view (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2484
2022-12-01T10:11:41.9903167Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2485
2022-12-01T10:11:41.9903442Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:41.9903566Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:41.9903845Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:41.9903978Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:41.9904142Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:11:41.9904409Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:41.9904518Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:41.9904853Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:41.9904988Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:41.9905189Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:11:41.9905375Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpojmse879
2022-12-01T10:11:41.9905560Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu7bp374m
2022-12-01T10:11:41.9905757Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpojmse879/_remote_module_non_scriptable.py
2022-12-01T10:11:41.9905946Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu7bp374m/_remote_module_non_scriptable.py
2022-12-01T10:11:41.9906103Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:11:41.9906276Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:11:41.9906345Z ok (1.855s)
2022-12-01T10:11:41.9906352Z 
2022-12-01T10:11:41.9906553Z ----------------------------------------------------------------------
2022-12-01T10:11:41.9906635Z Ran 1 test in 1.856s
2022-12-01T10:11:41.9906640Z 
2022-12-01T10:11:41.9906703Z OK
2022-12-01T10:11:41.9906708Z 
2022-12-01T10:11:41.9906793Z Generating XML reports...
2022-12-01T10:11:41.9907145Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221201100749.xml
2022-12-01T10:11:41.9907376Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:11:41.9907635Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:41.9907760Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:41.9908038Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:41.9908177Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:41.9908184Z 
2022-12-01T10:11:41.9908256Z Running tests...
2022-12-01T10:11:41.9908450Z ----------------------------------------------------------------------
2022-12-01T10:11:41.9908592Z   test_ignored_output (__main__.DistributedDataParallelTest)
2022-12-01T10:11:41.9908816Z Test that the output of a model can be ignored and that there is no ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2523
2022-12-01T10:11:41.9908958Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2524
2022-12-01T10:11:41.9909228Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:41.9909352Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:41.9909633Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:41.9909771Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:41.9909939Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:11:41.9910205Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:41.9910328Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:41.9910605Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:41.9910723Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:41.9910885Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:11:41.9911070Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjp376azg
2022-12-01T10:11:41.9911341Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbylnuwyh
2022-12-01T10:11:41.9911539Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjp376azg/_remote_module_non_scriptable.py
2022-12-01T10:11:41.9911762Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbylnuwyh/_remote_module_non_scriptable.py
2022-12-01T10:11:41.9911934Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:11:41.9912098Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:11:41.9912152Z ok (1.860s)
2022-12-01T10:11:41.9912157Z 
2022-12-01T10:11:41.9912356Z ----------------------------------------------------------------------
2022-12-01T10:11:41.9912433Z Ran 1 test in 1.860s
2022-12-01T10:11:41.9912438Z 
2022-12-01T10:11:41.9912500Z OK
2022-12-01T10:11:41.9912505Z 
2022-12-01T10:11:41.9912590Z Generating XML reports...
2022-12-01T10:11:41.9912943Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221201100752.xml
2022-12-01T10:11:41.9913177Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:11:41.9913451Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:41.9913577Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:41.9913841Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:41.9913977Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:41.9913982Z 
2022-12-01T10:11:41.9914057Z Running tests...
2022-12-01T10:11:41.9914252Z ----------------------------------------------------------------------
2022-12-01T10:11:41.9914420Z   test_ignored_output_with_unused_parameters (__main__.DistributedDataParallelTest)
2022-12-01T10:11:41.9914650Z Test that the output of a model can be ignored and that there is no ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2568
2022-12-01T10:11:41.9914809Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2569
2022-12-01T10:11:41.9915082Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:41.9915192Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:41.9915470Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:41.9915604Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:41.9915767Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:11:41.9916040Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:41.9916165Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:41.9916447Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:41.9916588Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:41.9916751Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:11:41.9916928Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplpkap2f2
2022-12-01T10:11:41.9917114Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpptiinwkq
2022-12-01T10:11:41.9917311Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplpkap2f2/_remote_module_non_scriptable.py
2022-12-01T10:11:41.9917503Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpptiinwkq/_remote_module_non_scriptable.py
2022-12-01T10:11:41.9917574Z ok (1.858s)
2022-12-01T10:11:41.9917579Z 
2022-12-01T10:11:41.9917812Z ----------------------------------------------------------------------
2022-12-01T10:11:41.9917891Z Ran 1 test in 1.858s
2022-12-01T10:11:41.9917924Z 
2022-12-01T10:11:41.9917988Z OK
2022-12-01T10:11:41.9917993Z 
2022-12-01T10:11:41.9918064Z Generating XML reports...
2022-12-01T10:11:41.9918416Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221201100756.xml
2022-12-01T10:11:41.9918648Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:11:41.9918925Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:41.9919051Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:41.9919336Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:41.9919473Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:41.9919478Z 
2022-12-01T10:11:41.9919555Z Running tests...
2022-12-01T10:11:41.9919755Z ----------------------------------------------------------------------
2022-12-01T10:11:41.9919989Z   test_invalid_powerSGD_state (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2613
2022-12-01T10:11:41.9920147Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2614
2022-12-01T10:11:41.9920419Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:41.9920543Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:41.9920824Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:41.9920960Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:41.9921127Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:11:41.9921539Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-12-01T10:11:41.9921955Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-12-01T10:11:41.9922359Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-12-01T10:11:41.9922766Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-12-01T10:11:41.9923165Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-12-01T10:11:41.9923600Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-12-01T10:11:41.9923903Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:41.9924026Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:41.9924295Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:41.9924433Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:41.9924595Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:11:41.9925001Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-12-01T10:11:41.9925411Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-12-01T10:11:41.9925807Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-12-01T10:11:41.9926213Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-12-01T10:11:41.9926615Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-12-01T10:11:41.9927015Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-12-01T10:11:41.9927085Z ok (1.753s)
2022-12-01T10:11:41.9927090Z 
2022-12-01T10:11:41.9927291Z ----------------------------------------------------------------------
2022-12-01T10:11:41.9927368Z Ran 1 test in 1.753s
2022-12-01T10:11:41.9927374Z 
2022-12-01T10:11:41.9927435Z OK
2022-12-01T10:11:41.9927440Z 
2022-12-01T10:11:41.9927523Z Generating XML reports...
2022-12-01T10:11:41.9927859Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221201100759.xml
2022-12-01T10:11:41.9928090Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:11:41.9928364Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:41.9928487Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:41.9928773Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:41.9928940Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:41.9928946Z 
2022-12-01T10:11:41.9929054Z Running tests...
2022-12-01T10:11:41.9929249Z ----------------------------------------------------------------------
2022-12-01T10:11:41.9929494Z   test_save_load_checkpoint (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2646
2022-12-01T10:11:41.9929636Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2647
2022-12-01T10:11:41.9929908Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:41.9930031Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:41.9930310Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:41.9930446Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:41.9930614Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:11:41.9930886Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:41.9931009Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:41.9931284Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:41.9931404Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:41.9931567Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:11:41.9931673Z skip: Need at least 2 CUDA devices (1.762s)
2022-12-01T10:11:41.9931678Z 
2022-12-01T10:11:41.9931870Z ----------------------------------------------------------------------
2022-12-01T10:11:41.9931947Z Ran 1 test in 1.763s
2022-12-01T10:11:41.9931953Z 
2022-12-01T10:11:41.9932025Z OK (skipped=1)
2022-12-01T10:11:41.9932030Z 
2022-12-01T10:11:41.9932116Z Generating XML reports...
2022-12-01T10:11:41.9932466Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221201100802.xml
2022-12-01T10:11:41.9932681Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:11:41.9932958Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:41.9933081Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:41.9933360Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:41.9933496Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:41.9933502Z 
2022-12-01T10:11:41.9933576Z Running tests...
2022-12-01T10:11:41.9933772Z ----------------------------------------------------------------------
2022-12-01T10:11:41.9934012Z   test_sparse_gradients (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2679
2022-12-01T10:11:41.9934167Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2680
2022-12-01T10:11:41.9934424Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:41.9934546Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:41.9934825Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:41.9934959Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:41.9935123Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:11:41.9935391Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:41.9935560Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:41.9935864Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:41.9935984Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:41.9936148Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:11:41.9936332Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7v843bs6
2022-12-01T10:11:41.9936516Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqx2ufeqq
2022-12-01T10:11:41.9936710Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7v843bs6/_remote_module_non_scriptable.py
2022-12-01T10:11:41.9936905Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqx2ufeqq/_remote_module_non_scriptable.py
2022-12-01T10:11:41.9936975Z ok (1.955s)
2022-12-01T10:11:41.9936981Z 
2022-12-01T10:11:41.9937180Z ----------------------------------------------------------------------
2022-12-01T10:11:41.9937256Z Ran 1 test in 1.955s
2022-12-01T10:11:41.9937262Z 
2022-12-01T10:11:41.9937309Z OK
2022-12-01T10:11:41.9937314Z 
2022-12-01T10:11:41.9937398Z Generating XML reports...
2022-12-01T10:11:41.9937744Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221201100806.xml
2022-12-01T10:11:41.9937977Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:11:41.9938248Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:41.9938457Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:41.9938746Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:41.9938885Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:41.9938891Z 
2022-12-01T10:11:41.9938952Z Running tests...
2022-12-01T10:11:41.9939148Z ----------------------------------------------------------------------
2022-12-01T10:11:41.9939396Z   test_sparse_gradients_grad_is_view (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2724
2022-12-01T10:11:41.9939551Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2725
2022-12-01T10:11:41.9939820Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:41.9939942Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:41.9940225Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:41.9940490Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:41.9940661Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:11:41.9940921Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:41.9941043Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:41.9941320Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:41.9941452Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:41.9941616Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:11:41.9941801Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsh38mk5x
2022-12-01T10:11:41.9941985Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2y4hx1o5
2022-12-01T10:11:41.9942235Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsh38mk5x/_remote_module_non_scriptable.py
2022-12-01T10:11:41.9942464Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2y4hx1o5/_remote_module_non_scriptable.py
2022-12-01T10:11:41.9942520Z ok (1.856s)
2022-12-01T10:11:41.9942525Z 
2022-12-01T10:11:41.9942722Z ----------------------------------------------------------------------
2022-12-01T10:11:41.9942800Z Ran 1 test in 1.856s
2022-12-01T10:11:41.9942806Z 
2022-12-01T10:11:41.9942867Z OK
2022-12-01T10:11:41.9942873Z 
2022-12-01T10:11:41.9942957Z Generating XML reports...
2022-12-01T10:11:41.9943311Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221201100809.xml
2022-12-01T10:11:41.9943541Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:11:41.9943815Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:41.9943929Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:41.9944209Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:41.9944346Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:41.9944352Z 
2022-12-01T10:11:41.9944424Z Running tests...
2022-12-01T10:11:41.9944620Z ----------------------------------------------------------------------
2022-12-01T10:11:41.9944869Z   test_sync_batch_norm_empty_input (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2769
2022-12-01T10:11:41.9945026Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2770
2022-12-01T10:11:41.9945297Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:41.9945421Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:41.9945695Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:41.9945831Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:41.9945995Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:11:41.9946264Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:41.9946386Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:41.9946664Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:41.9946798Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:41.9946960Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:11:41.9947051Z skip: Need at least 2 CUDA devices (1.860s)
2022-12-01T10:11:41.9947072Z 
2022-12-01T10:11:41.9947254Z ----------------------------------------------------------------------
2022-12-01T10:11:41.9947332Z Ran 1 test in 1.860s
2022-12-01T10:11:41.9947337Z 
2022-12-01T10:11:41.9947407Z OK (skipped=1)
2022-12-01T10:11:41.9947412Z 
2022-12-01T10:11:41.9947496Z Generating XML reports...
2022-12-01T10:11:41.9947840Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221201100813.xml
2022-12-01T10:11:41.9948071Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:11:41.9948343Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:41.9948465Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:41.9948735Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:41.9948903Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:41.9948950Z 
2022-12-01T10:11:41.9949024Z Running tests...
2022-12-01T10:11:41.9949220Z ----------------------------------------------------------------------
2022-12-01T10:11:41.9949472Z   test_sync_batch_norm_only_empty_input (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2802
2022-12-01T10:11:41.9949628Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2803
2022-12-01T10:11:41.9949900Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:41.9950021Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:41.9950301Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:41.9950425Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:41.9950589Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:11:41.9950864Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:41.9950986Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:41.9951264Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:41.9951397Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:41.9951558Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:11:41.9951663Z skip: Need at least 2 CUDA devices (1.859s)
2022-12-01T10:11:41.9951668Z 
2022-12-01T10:11:41.9951849Z ----------------------------------------------------------------------
2022-12-01T10:11:41.9951925Z Ran 1 test in 1.860s
2022-12-01T10:11:41.9951930Z 
2022-12-01T10:11:41.9952004Z OK (skipped=1)
2022-12-01T10:11:41.9952009Z 
2022-12-01T10:11:41.9952095Z Generating XML reports...
2022-12-01T10:11:41.9952442Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221201100816.xml
2022-12-01T10:11:41.9952674Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:11:41.9952946Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:41.9953069Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:41.9953348Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:41.9953469Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:41.9953474Z 
2022-12-01T10:11:41.9953546Z Running tests...
2022-12-01T10:11:41.9953744Z ----------------------------------------------------------------------
2022-12-01T10:11:41.9954025Z   test_collectives (__main__.GlooProcessGroupWithDispatchedCollectivesTests) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2835
2022-12-01T10:11:41.9954299Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:41.9954425Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:41.9954706Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:41.9954842Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:41.9954992Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:11:41.9955170Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:11:41.9955502Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-12-01T10:11:41.9955598Z ok (1.768s)
2022-12-01T10:11:41.9955604Z 
2022-12-01T10:11:41.9955798Z ----------------------------------------------------------------------
2022-12-01T10:11:41.9955872Z Ran 1 test in 1.769s
2022-12-01T10:11:41.9955877Z 
2022-12-01T10:11:41.9955939Z OK
2022-12-01T10:11:41.9955944Z 
2022-12-01T10:11:41.9956026Z Generating XML reports...
2022-12-01T10:11:41.9956448Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-GlooProcessGroupWithDispatchedCollectivesTests-20221201100820.xml
2022-12-01T10:11:41.9956665Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:11:41.9956939Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:41.9957063Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:41.9957346Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:41.9957483Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:41.9957489Z 
2022-12-01T10:11:41.9957560Z Running tests...
2022-12-01T10:11:41.9957755Z ----------------------------------------------------------------------
2022-12-01T10:11:41.9957981Z   test_allgather_basics (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2860
2022-12-01T10:11:41.9958136Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2861
2022-12-01T10:11:41.9958275Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2862
2022-12-01T10:11:41.9958426Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2863
2022-12-01T10:11:41.9958701Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:41.9958826Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:41.9959107Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:41.9959242Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:41.9959413Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:11:41.9959685Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:41.9959793Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:41.9960073Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:41.9960206Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:41.9960370Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:11:41.9960643Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:41.9960764Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:41.9961040Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:41.9961172Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:41.9961334Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:11:41.9961587Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:41.9961707Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:41.9962012Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:41.9962145Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:41.9962337Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:11:41.9962404Z ok (2.061s)
2022-12-01T10:11:41.9962409Z 
2022-12-01T10:11:41.9962602Z ----------------------------------------------------------------------
2022-12-01T10:11:41.9962677Z Ran 1 test in 2.061s
2022-12-01T10:11:41.9962682Z 
2022-12-01T10:11:41.9962729Z OK
2022-12-01T10:11:41.9962734Z 
2022-12-01T10:11:41.9962817Z Generating XML reports...
2022-12-01T10:11:41.9963143Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221201100823.xml
2022-12-01T10:11:41.9963373Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:11:41.9963645Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:41.9963771Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:41.9964053Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:41.9964190Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:41.9964195Z 
2022-12-01T10:11:41.9964268Z Running tests...
2022-12-01T10:11:41.9964448Z ----------------------------------------------------------------------
2022-12-01T10:11:41.9964678Z   test_allgather_basics_cuda (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2927
2022-12-01T10:11:41.9964831Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2928
2022-12-01T10:11:41.9964983Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2929
2022-12-01T10:11:41.9965133Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2930
2022-12-01T10:11:41.9965405Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:41.9965531Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:41.9965811Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:41.9965933Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:41.9966095Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:11:41.9966363Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:41.9966486Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:41.9966761Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:41.9966896Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:41.9967062Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:11:41.9967329Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:41.9967451Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:41.9967711Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:41.9967844Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:41.9968004Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:11:41.9968267Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:41.9968388Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:41.9968690Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:41.9968851Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:41.9969012Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:11:41.9969104Z skip: Need at least 2 CUDA devices (2.065s)
2022-12-01T10:11:41.9969109Z 
2022-12-01T10:11:41.9969305Z ----------------------------------------------------------------------
2022-12-01T10:11:41.9969381Z Ran 1 test in 2.065s
2022-12-01T10:11:41.9969386Z 
2022-12-01T10:11:41.9969456Z OK (skipped=1)
2022-12-01T10:11:41.9969461Z 
2022-12-01T10:11:41.9969543Z Generating XML reports...
2022-12-01T10:11:41.9969865Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221201100827.xml
2022-12-01T10:11:41.9970096Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:11:41.9970369Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:41.9970496Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:41.9970764Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:41.9970899Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:41.9970904Z 
2022-12-01T10:11:41.9970977Z Running tests...
2022-12-01T10:11:41.9971171Z ----------------------------------------------------------------------
2022-12-01T10:11:41.9971399Z   test_allgather_checks (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2982
2022-12-01T10:11:41.9971553Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2983
2022-12-01T10:11:41.9971708Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2984
2022-12-01T10:11:41.9971862Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2985
2022-12-01T10:11:41.9972122Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:41.9972246Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:41.9972525Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:41.9972661Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:41.9972824Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:11:41.9973094Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:41.9973215Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:41.9973493Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:41.9973628Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:41.9973778Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:11:41.9974044Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:41.9974164Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:41.9974440Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:41.9974572Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:41.9974733Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:11:41.9975032Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:41.9975181Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:41.9975452Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:41.9975588Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:41.9975751Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:11:41.9975820Z ok (2.073s)
2022-12-01T10:11:41.9975826Z 
2022-12-01T10:11:41.9976022Z ----------------------------------------------------------------------
2022-12-01T10:11:41.9976098Z Ran 1 test in 2.074s
2022-12-01T10:11:41.9976103Z 
2022-12-01T10:11:41.9976164Z OK
2022-12-01T10:11:41.9976169Z 
2022-12-01T10:11:41.9976252Z Generating XML reports...
2022-12-01T10:11:41.9976578Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221201100830.xml
2022-12-01T10:11:41.9976793Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:11:41.9977070Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:41.9977194Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:41.9977474Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:41.9977609Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:41.9977614Z 
2022-12-01T10:11:41.9977689Z Running tests...
2022-12-01T10:11:41.9977886Z ----------------------------------------------------------------------
2022-12-01T10:11:41.9978122Z   test_allgather_coalesced_async (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3049
2022-12-01T10:11:41.9978264Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3050
2022-12-01T10:11:41.9978541Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3051
2022-12-01T10:11:41.9978702Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3052
2022-12-01T10:11:41.9978980Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:41.9979103Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:41.9979382Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:41.9979517Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:41.9979683Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:11:41.9979955Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:41.9980063Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:41.9980465Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:41.9980607Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:41.9980769Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:11:41.9981038Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:41.9981162Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:41.9981437Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:41.9981568Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:41.9981785Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:11:41.9982090Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:41.9982211Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:41.9982484Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:41.9982616Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:41.9982780Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:11:41.9982966Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:11:41.9983145Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:11:41.9983323Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:11:41.9983612Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:11:41.9983917Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:11:41.9984089Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-12-01T10:11:41.9984385Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:11:41.9984679Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:11:41.9985264Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:2442: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-12-01T10:11:41.9985420Z   "torch.distributed.all_gather_coalesced will be deprecated. If you must "
2022-12-01T10:11:41.9985978Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:2442: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-12-01T10:11:41.9986124Z   "torch.distributed.all_gather_coalesced will be deprecated. If you must "
2022-12-01T10:11:41.9986674Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:2442: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-12-01T10:11:41.9986822Z   "torch.distributed.all_gather_coalesced will be deprecated. If you must "
2022-12-01T10:11:41.9987380Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:2442: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-12-01T10:11:41.9987517Z   "torch.distributed.all_gather_coalesced will be deprecated. If you must "
2022-12-01T10:11:41.9987586Z ok (1.965s)
2022-12-01T10:11:41.9987594Z 
2022-12-01T10:11:41.9987789Z ----------------------------------------------------------------------
2022-12-01T10:11:41.9987866Z Ran 1 test in 1.965s
2022-12-01T10:11:41.9987871Z 
2022-12-01T10:11:41.9987933Z OK
2022-12-01T10:11:41.9987938Z 
2022-12-01T10:11:41.9988021Z Generating XML reports...
2022-12-01T10:11:41.9988346Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221201100834.xml
2022-12-01T10:11:41.9988611Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:11:41.9988871Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:41.9989024Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:41.9989303Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:41.9989440Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:41.9989446Z 
2022-12-01T10:11:41.9989518Z Running tests...
2022-12-01T10:11:41.9989715Z ----------------------------------------------------------------------
2022-12-01T10:11:41.9989953Z   test_allgather_coalesced_checks (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3116
2022-12-01T10:11:41.9990110Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3117
2022-12-01T10:11:41.9990263Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3118
2022-12-01T10:11:41.9990404Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3119
2022-12-01T10:11:41.9990679Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:41.9990803Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:41.9991087Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:41.9991225Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:41.9991395Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:11:41.9991665Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:41.9991790Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:41.9992071Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:41.9992194Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:41.9992359Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:11:41.9992627Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:41.9992748Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:41.9993023Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:41.9993155Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:41.9993316Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:11:41.9993584Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:41.9993694Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:41.9993970Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:41.9994103Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:41.9994264Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:11:41.9994826Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:2442: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-12-01T10:11:41.9994977Z   "torch.distributed.all_gather_coalesced will be deprecated. If you must "
2022-12-01T10:11:41.9995553Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:2442: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-12-01T10:11:41.9995733Z   "torch.distributed.all_gather_coalesced will be deprecated. If you must "
2022-12-01T10:11:41.9996286Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:2442: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-12-01T10:11:41.9996431Z   "torch.distributed.all_gather_coalesced will be deprecated. If you must "
2022-12-01T10:11:41.9996982Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:2442: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-12-01T10:11:41.9997136Z   "torch.distributed.all_gather_coalesced will be deprecated. If you must "
2022-12-01T10:11:41.9997191Z ok (1.965s)
2022-12-01T10:11:41.9997196Z 
2022-12-01T10:11:41.9997393Z ----------------------------------------------------------------------
2022-12-01T10:11:41.9997469Z Ran 1 test in 1.965s
2022-12-01T10:11:41.9997475Z 
2022-12-01T10:11:41.9997536Z OK
2022-12-01T10:11:41.9997541Z 
2022-12-01T10:11:41.9997623Z Generating XML reports...
2022-12-01T10:11:41.9998064Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221201100838.xml
2022-12-01T10:11:41.9998302Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:11:41.9998574Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:41.9998688Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:41.9998965Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:41.9999104Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:41.9999109Z 
2022-12-01T10:11:41.9999183Z Running tests...
2022-12-01T10:11:41.9999377Z ----------------------------------------------------------------------
2022-12-01T10:11:41.9999617Z   test_allgather_noncontiguous_input (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3183
2022-12-01T10:11:41.9999773Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3184
2022-12-01T10:11:41.9999926Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3185
2022-12-01T10:11:42.0000064Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3186
2022-12-01T10:11:42.0000341Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0000467Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0000748Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0000883Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0001046Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:11:42.0001315Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0001438Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0001715Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0001833Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0002033Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:11:42.0002326Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0002448Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0002723Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0002854Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0003015Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:11:42.0003280Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0003400Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0003662Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0003801Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0003964Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:11:42.0004032Z ok (2.063s)
2022-12-01T10:11:42.0004038Z 
2022-12-01T10:11:42.0004233Z ----------------------------------------------------------------------
2022-12-01T10:11:42.0004309Z Ran 1 test in 2.063s
2022-12-01T10:11:42.0004314Z 
2022-12-01T10:11:42.0004375Z OK
2022-12-01T10:11:42.0004380Z 
2022-12-01T10:11:42.0004463Z Generating XML reports...
2022-12-01T10:11:42.0004775Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221201100841.xml
2022-12-01T10:11:42.0005005Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:11:42.0005281Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0005406Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0005685Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0005819Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0005824Z 
2022-12-01T10:11:42.0005896Z Running tests...
2022-12-01T10:11:42.0006090Z ----------------------------------------------------------------------
2022-12-01T10:11:42.0006301Z   test_allgather_stress (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3250
2022-12-01T10:11:42.0006457Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3251
2022-12-01T10:11:42.0006610Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3252
2022-12-01T10:11:42.0006766Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3253
2022-12-01T10:11:42.0007045Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0007170Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0007452Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0007588Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0007752Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:11:42.0008011Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0008135Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0008444Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0008581Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0008797Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:11:42.0009067Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0009189Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0009464Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0009596Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0009746Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:11:42.0010016Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0010144Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0010424Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0010558Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0010718Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:11:42.0010786Z ok (3.468s)
2022-12-01T10:11:42.0010791Z 
2022-12-01T10:11:42.0010989Z ----------------------------------------------------------------------
2022-12-01T10:11:42.0011051Z Ran 1 test in 3.468s
2022-12-01T10:11:42.0011056Z 
2022-12-01T10:11:42.0011117Z OK
2022-12-01T10:11:42.0011122Z 
2022-12-01T10:11:42.0011207Z Generating XML reports...
2022-12-01T10:11:42.0011534Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221201100845.xml
2022-12-01T10:11:42.0011766Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:11:42.0012041Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0012167Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0012452Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0012574Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0012593Z 
2022-12-01T10:11:42.0012653Z Running tests...
2022-12-01T10:11:42.0012847Z ----------------------------------------------------------------------
2022-12-01T10:11:42.0013078Z   test_allgather_stress_cuda (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3341
2022-12-01T10:11:42.0013234Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3342
2022-12-01T10:11:42.0013390Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3343
2022-12-01T10:11:42.0013544Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3344
2022-12-01T10:11:42.0013817Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0013941Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0014209Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0014343Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0014507Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:11:42.0014773Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0014896Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0015222Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0015383Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0015546Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:11:42.0015813Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0015920Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0016192Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0016326Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0016486Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:11:42.0016753Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0016878Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0017151Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0017283Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0017430Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:11:42.0018616Z skip: Need at least 2 CUDA devices (1.965s)
2022-12-01T10:11:42.0018622Z 
2022-12-01T10:11:42.0018818Z ----------------------------------------------------------------------
2022-12-01T10:11:42.0018892Z Ran 1 test in 1.966s
2022-12-01T10:11:42.0018897Z 
2022-12-01T10:11:42.0018969Z OK (skipped=1)
2022-12-01T10:11:42.0018974Z 
2022-12-01T10:11:42.0019059Z Generating XML reports...
2022-12-01T10:11:42.0019389Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221201100850.xml
2022-12-01T10:11:42.0019623Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:11:42.0019897Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0020005Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0020285Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0020523Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0020529Z 
2022-12-01T10:11:42.0020603Z Running tests...
2022-12-01T10:11:42.0020801Z ----------------------------------------------------------------------
2022-12-01T10:11:42.0021030Z   test_allreduce_basics (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3396
2022-12-01T10:11:42.0021188Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3397
2022-12-01T10:11:42.0021341Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3398
2022-12-01T10:11:42.0021492Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3399
2022-12-01T10:11:42.0021750Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0021875Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0022155Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0022291Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0022454Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:11:42.0022786Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0022953Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0023233Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0023354Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0023515Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:11:42.0023784Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0023908Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0024184Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0024317Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0024481Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:11:42.0024751Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0024873Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0025136Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0025270Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0025431Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:11:42.0025502Z ok (2.065s)
2022-12-01T10:11:42.0025508Z 
2022-12-01T10:11:42.0025703Z ----------------------------------------------------------------------
2022-12-01T10:11:42.0025780Z Ran 1 test in 2.065s
2022-12-01T10:11:42.0025785Z 
2022-12-01T10:11:42.0025848Z OK
2022-12-01T10:11:42.0025855Z 
2022-12-01T10:11:42.0025941Z Generating XML reports...
2022-12-01T10:11:42.0026253Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221201100854.xml
2022-12-01T10:11:42.0026486Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:11:42.0026759Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0026883Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0027164Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0027297Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0027302Z 
2022-12-01T10:11:42.0027375Z Running tests...
2022-12-01T10:11:42.0027573Z ----------------------------------------------------------------------
2022-12-01T10:11:42.0027804Z   test_allreduce_basics_cuda (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3463
2022-12-01T10:11:42.0027949Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3464
2022-12-01T10:11:42.0028104Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3465
2022-12-01T10:11:42.0028253Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3466
2022-12-01T10:11:42.0028526Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0028650Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0028929Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0029063Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0029271Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:11:42.0029550Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0029675Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0029952Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0030085Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0030248Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:11:42.0030518Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0030643Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0030926Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0031062Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0031216Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:11:42.0031489Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0031612Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0031890Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0032024Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0032191Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:11:42.0032298Z skip: Need at least 2 CUDA devices (1.965s)
2022-12-01T10:11:42.0032304Z 
2022-12-01T10:11:42.0032505Z ----------------------------------------------------------------------
2022-12-01T10:11:42.0032568Z Ran 1 test in 1.965s
2022-12-01T10:11:42.0032575Z 
2022-12-01T10:11:42.0032651Z OK (skipped=1)
2022-12-01T10:11:42.0032655Z 
2022-12-01T10:11:42.0032740Z Generating XML reports...
2022-12-01T10:11:42.0033133Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221201100857.xml
2022-12-01T10:11:42.0033481Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:11:42.0033907Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0034464Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0035070Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0035667Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0035918Z 
2022-12-01T10:11:42.0036038Z Running tests...
2022-12-01T10:11:42.0036459Z ----------------------------------------------------------------------
2022-12-01T10:11:42.0037017Z   test_allreduce_basics_cuda_using_work_api (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3518
2022-12-01T10:11:42.0037611Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3519
2022-12-01T10:11:42.0038144Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3520
2022-12-01T10:11:42.0038642Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3521
2022-12-01T10:11:42.0039307Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0039803Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0040625Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0041200Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0041811Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:11:42.0042609Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0043150Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0043808Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0044346Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0044920Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:11:42.0045682Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0046274Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0047077Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0047593Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0048073Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:11:42.0048826Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0049348Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0050093Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0050692Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0051241Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:11:42.0051746Z skip: Need at least 2 CUDA devices (1.966s)
2022-12-01T10:11:42.0052007Z 
2022-12-01T10:11:42.0052381Z ----------------------------------------------------------------------
2022-12-01T10:11:42.0052796Z Ran 1 test in 1.966s
2022-12-01T10:11:42.0053000Z 
2022-12-01T10:11:42.0053123Z OK (skipped=1)
2022-12-01T10:11:42.0053310Z 
2022-12-01T10:11:42.0053461Z Generating XML reports...
2022-12-01T10:11:42.0054278Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221201100901.xml
2022-12-01T10:11:42.0055160Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:11:42.0056010Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0056608Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0057403Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0058020Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0058315Z 
2022-12-01T10:11:42.0058530Z Running tests...
2022-12-01T10:11:42.0059080Z ----------------------------------------------------------------------
2022-12-01T10:11:42.0059799Z   test_allreduce_basics_using_work_api (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3573
2022-12-01T10:11:42.0060644Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3574
2022-12-01T10:11:42.0061254Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3575
2022-12-01T10:11:42.0061857Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3576
2022-12-01T10:11:42.0062670Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0063403Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0064251Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0064855Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0065438Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:11:42.0066298Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0066844Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0067515Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0068103Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0068681Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:11:42.0069538Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0070155Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0070931Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0071556Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0072137Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:11:42.0072968Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0073564Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0074348Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0074979Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0075550Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:11:42.0076013Z ok (2.066s)
2022-12-01T10:11:42.0076211Z 
2022-12-01T10:11:42.0076571Z ----------------------------------------------------------------------
2022-12-01T10:11:42.0076999Z Ran 1 test in 2.066s
2022-12-01T10:11:42.0077196Z 
2022-12-01T10:11:42.0077302Z OK
2022-12-01T10:11:42.0077465Z 
2022-12-01T10:11:42.0077613Z Generating XML reports...
2022-12-01T10:11:42.0078423Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221201100904.xml
2022-12-01T10:11:42.0079291Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:11:42.0080142Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0080737Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0081511Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0082105Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0082401Z 
2022-12-01T10:11:42.0082533Z Running tests...
2022-12-01T10:11:42.0123010Z ----------------------------------------------------------------------
2022-12-01T10:11:42.0123766Z   test_allreduce_checks (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3640
2022-12-01T10:11:42.0124495Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3641
2022-12-01T10:11:42.0125104Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3642
2022-12-01T10:11:42.0125707Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3643
2022-12-01T10:11:42.0126709Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0127387Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0128193Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0128829Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0129409Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:11:42.0130269Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0130866Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0131628Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0132262Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0132848Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:11:42.0133605Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0134160Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0134960Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0135514Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0135985Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:11:42.0136731Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0137200Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0137849Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0138353Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0138897Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:11:42.0139363Z ok (1.966s)
2022-12-01T10:11:42.0139547Z 
2022-12-01T10:11:42.0139892Z ----------------------------------------------------------------------
2022-12-01T10:11:42.0140213Z Ran 1 test in 1.967s
2022-12-01T10:11:42.0140484Z 
2022-12-01T10:11:42.0140575Z OK
2022-12-01T10:11:42.0140720Z 
2022-12-01T10:11:42.0140873Z Generating XML reports...
2022-12-01T10:11:42.0141591Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221201100908.xml
2022-12-01T10:11:42.0142390Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:11:42.0143101Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0143624Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0144314Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0144884Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0145159Z 
2022-12-01T10:11:42.0145266Z Running tests...
2022-12-01T10:11:42.0145752Z ----------------------------------------------------------------------
2022-12-01T10:11:42.0146312Z   test_allreduce_coalesced_async (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3707
2022-12-01T10:11:42.0146858Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3708
2022-12-01T10:11:42.0147196Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3709
2022-12-01T10:11:42.0147633Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3710
2022-12-01T10:11:42.0148161Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0148494Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0148924Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0149310Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0149635Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:11:42.0150100Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0150427Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0150842Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0151190Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0151511Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:11:42.0151960Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0152287Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0152713Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0153055Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0153363Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:11:42.0153821Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0154149Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0154560Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0154903Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0155243Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:11:42.0155601Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:11:42.0155965Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:11:42.0156335Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:11:42.0156701Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-12-01T10:11:42.0157194Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:11:42.0157703Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:11:42.0158219Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:11:42.0158727Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:11:42.0159522Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1578: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-12-01T10:11:42.0160066Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2022-12-01T10:11:42.0160835Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1578: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-12-01T10:11:42.0161398Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2022-12-01T10:11:42.0162137Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1578: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-12-01T10:11:42.0162669Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2022-12-01T10:11:42.0163384Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1578: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-12-01T10:11:42.0163910Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2022-12-01T10:11:42.0164148Z ok (2.164s)
2022-12-01T10:11:42.0164251Z 
2022-12-01T10:11:42.0164450Z ----------------------------------------------------------------------
2022-12-01T10:11:42.0164678Z Ran 1 test in 2.165s
2022-12-01T10:11:42.0164791Z 
2022-12-01T10:11:42.0164855Z OK
2022-12-01T10:11:42.0164948Z 
2022-12-01T10:11:42.0165030Z Generating XML reports...
2022-12-01T10:11:42.0165461Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221201100912.xml
2022-12-01T10:11:42.0165947Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:11:42.0166410Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0166746Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0167168Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0167514Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0167678Z 
2022-12-01T10:11:42.0167750Z Running tests...
2022-12-01T10:11:42.0168036Z ----------------------------------------------------------------------
2022-12-01T10:11:42.0168442Z   test_allreduce_coalesced_basics (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3774
2022-12-01T10:11:42.0168840Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3775
2022-12-01T10:11:42.0169171Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3776
2022-12-01T10:11:42.0169482Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3777
2022-12-01T10:11:42.0169928Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0170267Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0170683Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0171027Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0171436Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:11:42.0171903Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0172248Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0172673Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0173058Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0173370Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:11:42.0173864Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0174190Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0174618Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0174951Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0175273Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:11:42.0175727Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0176050Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0176462Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0176808Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0177129Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:11:42.0177366Z ok (2.065s)
2022-12-01T10:11:42.0177486Z 
2022-12-01T10:11:42.0177680Z ----------------------------------------------------------------------
2022-12-01T10:11:42.0177918Z Ran 1 test in 2.065s
2022-12-01T10:11:42.0178029Z 
2022-12-01T10:11:42.0178089Z OK
2022-12-01T10:11:42.0178168Z 
2022-12-01T10:11:42.0178250Z Generating XML reports...
2022-12-01T10:11:42.0178795Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221201100916.xml
2022-12-01T10:11:42.0179277Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:11:42.0179728Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0180061Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0180646Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0180996Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0181148Z 
2022-12-01T10:11:42.0181219Z Running tests...
2022-12-01T10:11:42.0181522Z ----------------------------------------------------------------------
2022-12-01T10:11:42.0181923Z   test_allreduce_coalesced_checks (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3841
2022-12-01T10:11:42.0182310Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3842
2022-12-01T10:11:42.0182638Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3843
2022-12-01T10:11:42.0182961Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3844
2022-12-01T10:11:42.0183419Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0183733Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0184160Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0184502Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0184811Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:11:42.0185278Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0185600Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0186125Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0186492Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0186817Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:11:42.0187274Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0187598Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0188009Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0188350Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0188672Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:11:42.0189116Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0189440Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0189865Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0190210Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0190516Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:11:42.0190768Z ok (2.163s)
2022-12-01T10:11:42.0190870Z 
2022-12-01T10:11:42.0191063Z ----------------------------------------------------------------------
2022-12-01T10:11:42.0191287Z Ran 1 test in 2.164s
2022-12-01T10:11:42.0191398Z 
2022-12-01T10:11:42.0191459Z OK
2022-12-01T10:11:42.0191551Z 
2022-12-01T10:11:42.0191635Z Generating XML reports...
2022-12-01T10:11:42.0192079Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221201100919.xml
2022-12-01T10:11:42.0192549Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:11:42.0193008Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0193339Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0193752Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0194098Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0194261Z 
2022-12-01T10:11:42.0194332Z Running tests...
2022-12-01T10:11:42.0194635Z ----------------------------------------------------------------------
2022-12-01T10:11:42.0195025Z   test_allreduce_coalesced_checks_cuda (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3908
2022-12-01T10:11:42.0195435Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3909
2022-12-01T10:11:42.0195799Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3910
2022-12-01T10:11:42.0196115Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3911
2022-12-01T10:11:42.0196564Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0196893Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0197318Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0197648Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0197968Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:11:42.0198424Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0198982Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0199623Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0199971Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0200291Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:11:42.0200736Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0201065Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0201491Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0201830Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0202138Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:11:42.0202597Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0202925Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0203337Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0203681Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0204002Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:11:42.0204284Z skip: Need at least 1 CUDA device (1.965s)
2022-12-01T10:11:42.0204417Z 
2022-12-01T10:11:42.0204599Z ----------------------------------------------------------------------
2022-12-01T10:11:42.0204839Z Ran 1 test in 1.965s
2022-12-01T10:11:42.0204950Z 
2022-12-01T10:11:42.0205021Z OK (skipped=1)
2022-12-01T10:11:42.0205126Z 
2022-12-01T10:11:42.0205197Z Generating XML reports...
2022-12-01T10:11:42.0205641Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221201100923.xml
2022-12-01T10:11:42.0206124Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:11:42.0206585Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0206897Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0207326Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0207671Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0207835Z 
2022-12-01T10:11:42.0207905Z Running tests...
2022-12-01T10:11:42.0208192Z ----------------------------------------------------------------------
2022-12-01T10:11:42.0208595Z   test_allreduce_coalesced_stress (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3963
2022-12-01T10:11:42.0208995Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3964
2022-12-01T10:11:42.0209311Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3965
2022-12-01T10:11:42.0209628Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3966
2022-12-01T10:11:42.0210073Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0210399Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0210812Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0211156Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0211475Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:11:42.0211958Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0212314Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0212736Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0213078Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0213386Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:11:42.0213839Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0214163Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0214587Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0214920Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0215242Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:11:42.0215697Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0216010Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0216438Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0216780Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0217103Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:11:42.0217341Z ok (2.866s)
2022-12-01T10:11:42.0217442Z 
2022-12-01T10:11:42.0217637Z ----------------------------------------------------------------------
2022-12-01T10:11:42.0217876Z Ran 1 test in 2.866s
2022-12-01T10:11:42.0217991Z 
2022-12-01T10:11:42.0218040Z OK
2022-12-01T10:11:42.0218131Z 
2022-12-01T10:11:42.0218216Z Generating XML reports...
2022-12-01T10:11:42.0218753Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221201100927.xml
2022-12-01T10:11:42.0219237Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:11:42.0219683Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0220009Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0220559Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0220898Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0221061Z 
2022-12-01T10:11:42.0221134Z Running tests...
2022-12-01T10:11:42.0221443Z ----------------------------------------------------------------------
2022-12-01T10:11:42.0221833Z   test_allreduce_stress (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4054
2022-12-01T10:11:42.0222215Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4055
2022-12-01T10:11:42.0222542Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4056
2022-12-01T10:11:42.0222866Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4057
2022-12-01T10:11:42.0223303Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0223631Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0224057Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0224399Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0224782Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:11:42.0225274Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0225600Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0226029Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0226360Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0226684Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:11:42.0227141Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0227453Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0227885Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0228229Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0228552Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:11:42.0228994Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0229320Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0229744Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0230072Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0230390Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:11:42.0230642Z ok (2.665s)
2022-12-01T10:11:42.0230745Z 
2022-12-01T10:11:42.0230944Z ----------------------------------------------------------------------
2022-12-01T10:11:42.0231172Z Ran 1 test in 2.666s
2022-12-01T10:11:42.0231285Z 
2022-12-01T10:11:42.0231346Z OK
2022-12-01T10:11:42.0231437Z 
2022-12-01T10:11:42.0231520Z Generating XML reports...
2022-12-01T10:11:42.0231949Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221201100931.xml
2022-12-01T10:11:42.0232430Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:11:42.0232886Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0233214Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0233624Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0233970Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0234133Z 
2022-12-01T10:11:42.0234207Z Running tests...
2022-12-01T10:11:42.0234494Z ----------------------------------------------------------------------
2022-12-01T10:11:42.0234886Z   test_allreduce_stress_cuda (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4145
2022-12-01T10:11:42.0235281Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4146
2022-12-01T10:11:42.0235606Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4147
2022-12-01T10:11:42.0235917Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4148
2022-12-01T10:11:42.0236360Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0236685Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0237150Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0237483Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0237837Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:11:42.0238288Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0238600Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0239026Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0239371Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0239698Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:11:42.0240138Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0240463Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0240889Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0241219Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0241537Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:11:42.0241991Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0242317Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0242726Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0243069Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0243391Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:11:42.0243667Z skip: Need at least 2 CUDA devices (1.867s)
2022-12-01T10:11:42.0243805Z 
2022-12-01T10:11:42.0244001Z ----------------------------------------------------------------------
2022-12-01T10:11:42.0244238Z Ran 1 test in 1.867s
2022-12-01T10:11:42.0244350Z 
2022-12-01T10:11:42.0244420Z OK (skipped=1)
2022-12-01T10:11:42.0244514Z 
2022-12-01T10:11:42.0244597Z Generating XML reports...
2022-12-01T10:11:42.0245038Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221201100935.xml
2022-12-01T10:11:42.0245512Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:11:42.0245957Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0246284Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0246713Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0247058Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0247219Z 
2022-12-01T10:11:42.0247277Z Running tests...
2022-12-01T10:11:42.0247574Z ----------------------------------------------------------------------
2022-12-01T10:11:42.0247965Z   test_barrier_implies_wait (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4200
2022-12-01T10:11:42.0248356Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4201
2022-12-01T10:11:42.0248673Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4202
2022-12-01T10:11:42.0248993Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4203
2022-12-01T10:11:42.0249440Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0249790Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0250216Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0250591Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0250911Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:11:42.0251352Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0251678Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0252102Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0252431Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0252752Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:11:42.0253207Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0253532Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0253942Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0254285Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0254613Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:11:42.0255069Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0255379Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0255799Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0256144Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0256452Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:11:42.0256706Z ok (2.063s)
2022-12-01T10:11:42.0256808Z 
2022-12-01T10:11:42.0257004Z ----------------------------------------------------------------------
2022-12-01T10:11:42.0257238Z Ran 1 test in 2.064s
2022-12-01T10:11:42.0257339Z 
2022-12-01T10:11:42.0257399Z OK
2022-12-01T10:11:42.0257490Z 
2022-12-01T10:11:42.0257573Z Generating XML reports...
2022-12-01T10:11:42.0258013Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221201100939.xml
2022-12-01T10:11:42.0258562Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:11:42.0259027Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0259362Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0259792Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0260127Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0260290Z 
2022-12-01T10:11:42.0260535Z Running tests...
2022-12-01T10:11:42.0260853Z ----------------------------------------------------------------------
2022-12-01T10:11:42.0261231Z   test_broadcast_basics (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4267
2022-12-01T10:11:42.0261619Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4268
2022-12-01T10:11:42.0261947Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4269
2022-12-01T10:11:42.0262274Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4270
2022-12-01T10:11:42.0262814Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0263183Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0263616Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0263947Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0264270Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:11:42.0264728Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0265057Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0265472Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0265816Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0266140Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:11:42.0266599Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0266915Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0267404Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0267745Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0268056Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:11:42.0268516Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0268845Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0269272Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0269609Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0269931Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:11:42.0270182Z ok (1.965s)
2022-12-01T10:11:42.0270286Z 
2022-12-01T10:11:42.0270471Z ----------------------------------------------------------------------
2022-12-01T10:11:42.0270710Z Ran 1 test in 1.965s
2022-12-01T10:11:42.0270825Z 
2022-12-01T10:11:42.0270884Z OK
2022-12-01T10:11:42.0270974Z 
2022-12-01T10:11:42.0271058Z Generating XML reports...
2022-12-01T10:11:42.0271488Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221201100942.xml
2022-12-01T10:11:42.0271969Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:11:42.0272434Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0272755Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0273184Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0273531Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0273691Z 
2022-12-01T10:11:42.0273762Z Running tests...
2022-12-01T10:11:42.0274051Z ----------------------------------------------------------------------
2022-12-01T10:11:42.0274455Z   test_broadcast_basics_cuda (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4334
2022-12-01T10:11:42.0274853Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4335
2022-12-01T10:11:42.0275183Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4336
2022-12-01T10:11:42.0275548Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4337
2022-12-01T10:11:42.0276025Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0276384Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0276797Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0277145Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0277471Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:11:42.0277931Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0278246Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0278684Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0279025Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0279340Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:11:42.0279798Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0280124Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0280546Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0280876Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0281199Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:11:42.0281657Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0281986Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0282398Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0282746Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0283070Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:11:42.0283344Z skip: Need at least 2 CUDA devices (1.962s)
2022-12-01T10:11:42.0283484Z 
2022-12-01T10:11:42.0283677Z ----------------------------------------------------------------------
2022-12-01T10:11:42.0283919Z Ran 1 test in 1.962s
2022-12-01T10:11:42.0284030Z 
2022-12-01T10:11:42.0284101Z OK (skipped=1)
2022-12-01T10:11:42.0284195Z 
2022-12-01T10:11:42.0284284Z Generating XML reports...
2022-12-01T10:11:42.0284727Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221201100946.xml
2022-12-01T10:11:42.0285216Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:11:42.0285667Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0285998Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0286425Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0286777Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0286941Z 
2022-12-01T10:11:42.0286999Z Running tests...
2022-12-01T10:11:42.0287201Z ----------------------------------------------------------------------
2022-12-01T10:11:42.0287429Z   test_broadcast_checks (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4389
2022-12-01T10:11:42.0287589Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4390
2022-12-01T10:11:42.0287781Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4391
2022-12-01T10:11:42.0287970Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4392
2022-12-01T10:11:42.0288245Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0288368Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0288635Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0288773Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0288944Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:11:42.0289216Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0289341Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0289615Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0289754Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0289921Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:11:42.0290177Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0290298Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0290571Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0290707Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0290871Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:11:42.0291139Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0291262Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0291534Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0291666Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0291812Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:11:42.0291884Z ok (2.069s)
2022-12-01T10:11:42.0291889Z 
2022-12-01T10:11:42.0292085Z ----------------------------------------------------------------------
2022-12-01T10:11:42.0292160Z Ran 1 test in 2.069s
2022-12-01T10:11:42.0292165Z 
2022-12-01T10:11:42.0292228Z OK
2022-12-01T10:11:42.0292233Z 
2022-12-01T10:11:42.0292317Z Generating XML reports...
2022-12-01T10:11:42.0292647Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221201100950.xml
2022-12-01T10:11:42.0292880Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:11:42.0293138Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0293264Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0293543Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0293678Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0293683Z 
2022-12-01T10:11:42.0293754Z Running tests...
2022-12-01T10:11:42.0293948Z ----------------------------------------------------------------------
2022-12-01T10:11:42.0294209Z   test_broadcast_stress (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4456
2022-12-01T10:11:42.0294368Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4457
2022-12-01T10:11:42.0294551Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4458
2022-12-01T10:11:42.0294689Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4459
2022-12-01T10:11:42.0294961Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0295086Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0295367Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0295501Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0295664Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:11:42.0295938Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0296061Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0296340Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0296458Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0296620Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:11:42.0296890Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0297012Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0297288Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0297419Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0297581Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:11:42.0297850Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0297957Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0298230Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0298452Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0298625Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:11:42.0298694Z ok (2.364s)
2022-12-01T10:11:42.0298699Z 
2022-12-01T10:11:42.0298901Z ----------------------------------------------------------------------
2022-12-01T10:11:42.0298979Z Ran 1 test in 2.364s
2022-12-01T10:11:42.0298984Z 
2022-12-01T10:11:42.0299048Z OK
2022-12-01T10:11:42.0299052Z 
2022-12-01T10:11:42.0299141Z Generating XML reports...
2022-12-01T10:11:42.0299457Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221201100953.xml
2022-12-01T10:11:42.0299692Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:11:42.0299967Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0300090Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0300598Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0300739Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0300745Z 
2022-12-01T10:11:42.0300820Z Running tests...
2022-12-01T10:11:42.0301017Z ----------------------------------------------------------------------
2022-12-01T10:11:42.0301296Z   test_broadcast_stress_cuda (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4547
2022-12-01T10:11:42.0301492Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4548
2022-12-01T10:11:42.0301647Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4549
2022-12-01T10:11:42.0301801Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4550
2022-12-01T10:11:42.0302076Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0302202Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0302489Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0302625Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0302796Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:11:42.0303055Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0303177Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0303455Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0303589Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0303751Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:11:42.0304018Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0304143Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0304420Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0304550Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0304701Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:11:42.0304968Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0305089Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0305362Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0305494Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0305651Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:11:42.0305753Z skip: Need at least 2 CUDA devices (1.967s)
2022-12-01T10:11:42.0305758Z 
2022-12-01T10:11:42.0305956Z ----------------------------------------------------------------------
2022-12-01T10:11:42.0306018Z Ran 1 test in 1.967s
2022-12-01T10:11:42.0306040Z 
2022-12-01T10:11:42.0306099Z OK (skipped=1)
2022-12-01T10:11:42.0306103Z 
2022-12-01T10:11:42.0306186Z Generating XML reports...
2022-12-01T10:11:42.0306509Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221201100957.xml
2022-12-01T10:11:42.0306744Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:11:42.0307018Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0307141Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0307420Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0307556Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0307561Z 
2022-12-01T10:11:42.0307651Z Running tests...
2022-12-01T10:11:42.0307849Z ----------------------------------------------------------------------
2022-12-01T10:11:42.0308100Z   test_empty_tensors (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4602
2022-12-01T10:11:42.0308254Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4603
2022-12-01T10:11:42.0308409Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4604
2022-12-01T10:11:42.0308561Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4605
2022-12-01T10:11:42.0308833Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0308956Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0309233Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0309356Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0309524Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:11:42.0309793Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0309915Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0310198Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0310337Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0310500Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:11:42.0310777Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0310889Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0311165Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0311300Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0311462Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:11:42.0311726Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0311848Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0312123Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0312252Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0312414Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:11:42.0312471Z ok (2.066s)
2022-12-01T10:11:42.0312477Z 
2022-12-01T10:11:42.0312671Z ----------------------------------------------------------------------
2022-12-01T10:11:42.0312746Z Ran 1 test in 2.066s
2022-12-01T10:11:42.0312751Z 
2022-12-01T10:11:42.0312811Z OK
2022-12-01T10:11:42.0312816Z 
2022-12-01T10:11:42.0312899Z Generating XML reports...
2022-12-01T10:11:42.0313225Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221201101001.xml
2022-12-01T10:11:42.0313458Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:11:42.0313735Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0313845Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0314127Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0314293Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0314326Z 
2022-12-01T10:11:42.0314400Z Running tests...
2022-12-01T10:11:42.0314601Z ----------------------------------------------------------------------
2022-12-01T10:11:42.0314822Z   test_gather_basics (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4669
2022-12-01T10:11:42.0314975Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4670
2022-12-01T10:11:42.0315129Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4671
2022-12-01T10:11:42.0315279Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4672
2022-12-01T10:11:42.0315543Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0315666Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0315953Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0316093Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0316265Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:11:42.0316537Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0316660Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0316941Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0317064Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0317227Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:11:42.0317502Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0317627Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0317907Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0318043Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0318203Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:11:42.0318469Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0318590Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0318850Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0318983Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0319145Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:11:42.0319215Z ok (2.071s)
2022-12-01T10:11:42.0319220Z 
2022-12-01T10:11:42.0319424Z ----------------------------------------------------------------------
2022-12-01T10:11:42.0319499Z Ran 1 test in 2.071s
2022-12-01T10:11:42.0319505Z 
2022-12-01T10:11:42.0319564Z OK
2022-12-01T10:11:42.0319570Z 
2022-12-01T10:11:42.0319653Z Generating XML reports...
2022-12-01T10:11:42.0319966Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221201101005.xml
2022-12-01T10:11:42.0320197Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:11:42.0320485Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0320609Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0320924Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0321091Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0321096Z 
2022-12-01T10:11:42.0321168Z Running tests...
2022-12-01T10:11:42.0321362Z ----------------------------------------------------------------------
2022-12-01T10:11:42.0321587Z   test_gather_basics_cuda (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4736
2022-12-01T10:11:42.0321729Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4737
2022-12-01T10:11:42.0321885Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4738
2022-12-01T10:11:42.0322034Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4739
2022-12-01T10:11:42.0322388Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0322521Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0322809Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0322947Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0323110Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:11:42.0323378Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0323485Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0323765Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0323901Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0324067Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:11:42.0324334Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0324460Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0324735Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0324870Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0325018Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:11:42.0325287Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0325408Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0325689Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0325821Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0325991Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:11:42.0326095Z skip: Need at least 2 CUDA devices (1.968s)
2022-12-01T10:11:42.0326100Z 
2022-12-01T10:11:42.0326294Z ----------------------------------------------------------------------
2022-12-01T10:11:42.0326372Z Ran 1 test in 1.969s
2022-12-01T10:11:42.0326377Z 
2022-12-01T10:11:42.0326435Z OK (skipped=1)
2022-12-01T10:11:42.0326440Z 
2022-12-01T10:11:42.0326526Z Generating XML reports...
2022-12-01T10:11:42.0326854Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221201101008.xml
2022-12-01T10:11:42.0327085Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:11:42.0327411Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0327536Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0327853Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0327990Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0327995Z 
2022-12-01T10:11:42.0328068Z Running tests...
2022-12-01T10:11:42.0328252Z ----------------------------------------------------------------------
2022-12-01T10:11:42.0328479Z   test_gather_checks (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4791
2022-12-01T10:11:42.0328639Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4792
2022-12-01T10:11:42.0328792Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4793
2022-12-01T10:11:42.0328948Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4794
2022-12-01T10:11:42.0329220Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0329346Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0329625Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0329747Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0329913Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:11:42.0330186Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0330309Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0330592Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0330731Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0330897Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:11:42.0331167Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0331288Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0331552Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0331689Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0331850Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:11:42.0332120Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0332243Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0332518Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0332655Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0332812Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:11:42.0332868Z ok (2.067s)
2022-12-01T10:11:42.0332885Z 
2022-12-01T10:11:42.0333068Z ----------------------------------------------------------------------
2022-12-01T10:11:42.0333146Z Ran 1 test in 2.067s
2022-12-01T10:11:42.0333151Z 
2022-12-01T10:11:42.0333212Z OK
2022-12-01T10:11:42.0333217Z 
2022-12-01T10:11:42.0333300Z Generating XML reports...
2022-12-01T10:11:42.0333623Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221201101012.xml
2022-12-01T10:11:42.0333855Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:11:42.0334159Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0334311Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0334580Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0334715Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0334720Z 
2022-12-01T10:11:42.0334792Z Running tests...
2022-12-01T10:11:42.0334989Z ----------------------------------------------------------------------
2022-12-01T10:11:42.0335220Z   test_gather_noncontiguous_input (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4858
2022-12-01T10:11:42.0335375Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4859
2022-12-01T10:11:42.0335529Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4860
2022-12-01T10:11:42.0335679Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4861
2022-12-01T10:11:42.0335953Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0336062Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0336341Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0336474Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0336638Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:11:42.0336910Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0337035Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0337314Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0337461Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0337608Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:11:42.0337878Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0338001Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0338274Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0338499Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0338668Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:11:42.0338937Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0339109Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0339384Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0339503Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0339671Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:11:42.0339741Z ok (2.075s)
2022-12-01T10:11:42.0339747Z 
2022-12-01T10:11:42.0339947Z ----------------------------------------------------------------------
2022-12-01T10:11:42.0340023Z Ran 1 test in 2.075s
2022-12-01T10:11:42.0340028Z 
2022-12-01T10:11:42.0340091Z OK
2022-12-01T10:11:42.0340096Z 
2022-12-01T10:11:42.0340181Z Generating XML reports...
2022-12-01T10:11:42.0340723Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221201101016.xml
2022-12-01T10:11:42.0340954Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:11:42.0341270Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0341395Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0341681Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0341817Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0341822Z 
2022-12-01T10:11:42.0341895Z Running tests...
2022-12-01T10:11:42.0342096Z ----------------------------------------------------------------------
2022-12-01T10:11:42.0342323Z   test_gather_stress (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4925
2022-12-01T10:11:42.0342481Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4926
2022-12-01T10:11:42.0342623Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4927
2022-12-01T10:11:42.0342773Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4928
2022-12-01T10:11:42.0343046Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0343166Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0343446Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0343582Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0343743Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:11:42.0344015Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0344125Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0344402Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0344535Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0344695Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:11:42.0344964Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0345086Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0345358Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0345491Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0345656Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:11:42.0345907Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0346034Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0346310Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0346444Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0346607Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:11:42.0346676Z ok (3.773s)
2022-12-01T10:11:42.0346680Z 
2022-12-01T10:11:42.0346873Z ----------------------------------------------------------------------
2022-12-01T10:11:42.0346947Z Ran 1 test in 3.773s
2022-12-01T10:11:42.0346952Z 
2022-12-01T10:11:42.0346999Z OK
2022-12-01T10:11:42.0347057Z 
2022-12-01T10:11:42.0347130Z Generating XML reports...
2022-12-01T10:11:42.0347494Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221201101019.xml
2022-12-01T10:11:42.0347753Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:11:42.0348026Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0348148Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0348429Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0348566Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0348570Z 
2022-12-01T10:11:42.0348642Z Running tests...
2022-12-01T10:11:42.0348823Z ----------------------------------------------------------------------
2022-12-01T10:11:42.0349049Z   test_gather_stress_cuda (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5016
2022-12-01T10:11:42.0349208Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5017
2022-12-01T10:11:42.0349363Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5018
2022-12-01T10:11:42.0349518Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5019
2022-12-01T10:11:42.0349789Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0349918Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0350200Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0350336Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0350484Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:11:42.0350760Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0350883Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0351166Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0351301Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0351461Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:11:42.0351729Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0351852Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0352112Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0352247Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0352412Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:11:42.0352682Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0352803Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0353080Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0353213Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0353370Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:11:42.0353473Z skip: Need at least 2 CUDA devices (1.965s)
2022-12-01T10:11:42.0353479Z 
2022-12-01T10:11:42.0353659Z ----------------------------------------------------------------------
2022-12-01T10:11:42.0353736Z Ran 1 test in 1.966s
2022-12-01T10:11:42.0353778Z 
2022-12-01T10:11:42.0353851Z OK (skipped=1)
2022-12-01T10:11:42.0353882Z 
2022-12-01T10:11:42.0353966Z Generating XML reports...
2022-12-01T10:11:42.0354297Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221201101025.xml
2022-12-01T10:11:42.0354533Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:11:42.0354807Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0354933Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0355213Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0355336Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0355341Z 
2022-12-01T10:11:42.0355417Z Running tests...
2022-12-01T10:11:42.0355614Z ----------------------------------------------------------------------
2022-12-01T10:11:42.0355853Z   test_multi_device_constructor (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5071
2022-12-01T10:11:42.0356009Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5072
2022-12-01T10:11:42.0356164Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5073
2022-12-01T10:11:42.0356314Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5074
2022-12-01T10:11:42.0356587Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0356696Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0356976Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0357119Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0357285Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:11:42.0357560Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0357681Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0357962Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0358098Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0358262Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:11:42.0358518Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0358644Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0358930Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0359069Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0359229Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:11:42.0359500Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0359624Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0359902Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0360022Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0360184Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:11:42.0360259Z ok (2.169s)
2022-12-01T10:11:42.0360264Z 
2022-12-01T10:11:42.0360492Z ----------------------------------------------------------------------
2022-12-01T10:11:42.0360594Z Ran 1 test in 2.169s
2022-12-01T10:11:42.0360600Z 
2022-12-01T10:11:42.0360659Z OK
2022-12-01T10:11:42.0360664Z 
2022-12-01T10:11:42.0360753Z Generating XML reports...
2022-12-01T10:11:42.0361074Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221201101028.xml
2022-12-01T10:11:42.0361306Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:11:42.0361566Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0361689Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0361966Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0362106Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0362112Z 
2022-12-01T10:11:42.0362188Z Running tests...
2022-12-01T10:11:42.0362382Z ----------------------------------------------------------------------
2022-12-01T10:11:42.0362606Z   test_reduce_basics (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5142
2022-12-01T10:11:42.0362764Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5143
2022-12-01T10:11:42.0362903Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5144
2022-12-01T10:11:42.0363053Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5145
2022-12-01T10:11:42.0363327Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0363451Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0363736Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0363870Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0364031Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:11:42.0364305Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0364427Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0364696Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0364833Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0364993Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:11:42.0365260Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0365381Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0365663Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0365793Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0365953Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:11:42.0366223Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0366332Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0366608Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0366741Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0366932Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:11:42.0367001Z ok (2.065s)
2022-12-01T10:11:42.0367032Z 
2022-12-01T10:11:42.0367226Z ----------------------------------------------------------------------
2022-12-01T10:11:42.0367301Z Ran 1 test in 2.065s
2022-12-01T10:11:42.0367306Z 
2022-12-01T10:11:42.0367363Z OK
2022-12-01T10:11:42.0367368Z 
2022-12-01T10:11:42.0367439Z Generating XML reports...
2022-12-01T10:11:42.0367762Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221201101032.xml
2022-12-01T10:11:42.0367993Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:11:42.0368266Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0368388Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0368669Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0368802Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0368807Z 
2022-12-01T10:11:42.0368877Z Running tests...
2022-12-01T10:11:42.0369074Z ----------------------------------------------------------------------
2022-12-01T10:11:42.0369288Z   test_reduce_basics_cuda (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5209
2022-12-01T10:11:42.0369442Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5210
2022-12-01T10:11:42.0369598Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5211
2022-12-01T10:11:42.0369747Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5212
2022-12-01T10:11:42.0370019Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0370143Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0370418Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0370551Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0370713Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:11:42.0370967Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0371091Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0371367Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0371499Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0371657Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:11:42.0371927Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0372047Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0372318Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0372436Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0372599Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:11:42.0372864Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0372981Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0373252Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0373425Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0373634Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:11:42.0373735Z skip: Need at least 2 CUDA devices (1.967s)
2022-12-01T10:11:42.0373741Z 
2022-12-01T10:11:42.0373938Z ----------------------------------------------------------------------
2022-12-01T10:11:42.0373999Z Ran 1 test in 1.967s
2022-12-01T10:11:42.0374004Z 
2022-12-01T10:11:42.0374074Z OK (skipped=1)
2022-12-01T10:11:42.0374079Z 
2022-12-01T10:11:42.0374162Z Generating XML reports...
2022-12-01T10:11:42.0374491Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221201101036.xml
2022-12-01T10:11:42.0374725Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:11:42.0374998Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0375123Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0375404Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0375542Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0375547Z 
2022-12-01T10:11:42.0375607Z Running tests...
2022-12-01T10:11:42.0375803Z ----------------------------------------------------------------------
2022-12-01T10:11:42.0376028Z   test_reduce_checks (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5264
2022-12-01T10:11:42.0376184Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5265
2022-12-01T10:11:42.0376333Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5266
2022-12-01T10:11:42.0376485Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5267
2022-12-01T10:11:42.0376759Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0376890Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0377155Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0377291Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0377457Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:11:42.0377728Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0377854Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0378127Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0378261Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0378509Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:11:42.0378790Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0378898Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0379173Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0379308Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0379470Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:11:42.0379735Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0379857Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0380176Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0380335Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0380608Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:11:42.0380667Z ok (2.078s)
2022-12-01T10:11:42.0380673Z 
2022-12-01T10:11:42.0380870Z ----------------------------------------------------------------------
2022-12-01T10:11:42.0380946Z Ran 1 test in 2.078s
2022-12-01T10:11:42.0380951Z 
2022-12-01T10:11:42.0381016Z OK
2022-12-01T10:11:42.0381021Z 
2022-12-01T10:11:42.0381107Z Generating XML reports...
2022-12-01T10:11:42.0381432Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221201101039.xml
2022-12-01T10:11:42.0381664Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:11:42.0381942Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0382055Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0382338Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0382475Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0382479Z 
2022-12-01T10:11:42.0382551Z Running tests...
2022-12-01T10:11:42.0382745Z ----------------------------------------------------------------------
2022-12-01T10:11:42.0382967Z   test_reduce_stress (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5331
2022-12-01T10:11:42.0383119Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5332
2022-12-01T10:11:42.0383274Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5333
2022-12-01T10:11:42.0383429Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5334
2022-12-01T10:11:42.0383685Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0383811Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0384093Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0384228Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0384391Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:11:42.0384662Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0384784Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0385066Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0385192Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0385353Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:11:42.0385623Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0385747Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0386024Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0386156Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0386320Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:11:42.0386588Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0386768Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0387062Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0387202Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0387369Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:11:42.0387441Z ok (3.671s)
2022-12-01T10:11:42.0387446Z 
2022-12-01T10:11:42.0387645Z ----------------------------------------------------------------------
2022-12-01T10:11:42.0387721Z Ran 1 test in 3.671s
2022-12-01T10:11:42.0387727Z 
2022-12-01T10:11:42.0387790Z OK
2022-12-01T10:11:42.0387794Z 
2022-12-01T10:11:42.0387890Z Generating XML reports...
2022-12-01T10:11:42.0388203Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221201101043.xml
2022-12-01T10:11:42.0388435Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:11:42.0388715Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0388846Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0389127Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0389263Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0389268Z 
2022-12-01T10:11:42.0389338Z Running tests...
2022-12-01T10:11:42.0389533Z ----------------------------------------------------------------------
2022-12-01T10:11:42.0389758Z   test_reduce_stress_cuda (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5422
2022-12-01T10:11:42.0389900Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5423
2022-12-01T10:11:42.0390058Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5424
2022-12-01T10:11:42.0390209Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5425
2022-12-01T10:11:42.0390484Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0390609Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0390890Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0391026Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0391192Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:11:42.0391464Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0391576Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0391864Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0392001Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0392171Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:11:42.0392443Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0392565Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0392843Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0392979Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0393130Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:11:42.0393439Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0393590Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0393869Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0394003Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0394166Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:11:42.0394274Z skip: Need at least 2 CUDA devices (1.964s)
2022-12-01T10:11:42.0394279Z 
2022-12-01T10:11:42.0394479Z ----------------------------------------------------------------------
2022-12-01T10:11:42.0394556Z Ran 1 test in 1.964s
2022-12-01T10:11:42.0394561Z 
2022-12-01T10:11:42.0394622Z OK (skipped=1)
2022-12-01T10:11:42.0394626Z 
2022-12-01T10:11:42.0394754Z Generating XML reports...
2022-12-01T10:11:42.0395083Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221201101048.xml
2022-12-01T10:11:42.0395316Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:11:42.0395589Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0395712Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0395992Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0396127Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0396132Z 
2022-12-01T10:11:42.0396194Z Running tests...
2022-12-01T10:11:42.0396388Z ----------------------------------------------------------------------
2022-12-01T10:11:42.0396605Z   test_round_robin (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5477
2022-12-01T10:11:42.0396763Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5478
2022-12-01T10:11:42.0396917Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5479
2022-12-01T10:11:42.0397071Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5480
2022-12-01T10:11:42.0397345Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0397469Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0397749Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0397870Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0398033Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:11:42.0398301Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0398426Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0398702Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0398835Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0398996Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:11:42.0399263Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0399370Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0399648Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0399818Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0399981Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:11:42.0400275Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0400396Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0400672Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0400807Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0400966Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:11:42.0401398Z [W ProcessGroupRoundRobin.cpp:12] Warning: ProcessGroupRoundRobin is deprecated and scheduled to be removed after this current release (1.13). Please file an issue on https://github.com/pytorch/pytorch/issues if there are any concerns or issues with this deprecation. (function ProcessGroupRoundRobin)
2022-12-01T10:11:42.0401816Z [W ProcessGroupRoundRobin.cpp:12] Warning: ProcessGroupRoundRobin is deprecated and scheduled to be removed after this current release (1.13). Please file an issue on https://github.com/pytorch/pytorch/issues if there are any concerns or issues with this deprecation. (function ProcessGroupRoundRobin)
2022-12-01T10:11:42.0402224Z [W ProcessGroupRoundRobin.cpp:12] Warning: ProcessGroupRoundRobin is deprecated and scheduled to be removed after this current release (1.13). Please file an issue on https://github.com/pytorch/pytorch/issues if there are any concerns or issues with this deprecation. (function ProcessGroupRoundRobin)
2022-12-01T10:11:42.0402632Z [W ProcessGroupRoundRobin.cpp:12] Warning: ProcessGroupRoundRobin is deprecated and scheduled to be removed after this current release (1.13). Please file an issue on https://github.com/pytorch/pytorch/issues if there are any concerns or issues with this deprecation. (function ProcessGroupRoundRobin)
2022-12-01T10:11:42.0402703Z ok (2.068s)
2022-12-01T10:11:42.0402711Z 
2022-12-01T10:11:42.0402910Z ----------------------------------------------------------------------
2022-12-01T10:11:42.0402987Z Ran 1 test in 2.068s
2022-12-01T10:11:42.0402993Z 
2022-12-01T10:11:42.0403054Z OK
2022-12-01T10:11:42.0403059Z 
2022-12-01T10:11:42.0403143Z Generating XML reports...
2022-12-01T10:11:42.0403477Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221201101052.xml
2022-12-01T10:11:42.0403712Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:11:42.0403984Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0404096Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0404380Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0404517Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0404524Z 
2022-12-01T10:11:42.0404600Z Running tests...
2022-12-01T10:11:42.0404798Z ----------------------------------------------------------------------
2022-12-01T10:11:42.0405028Z   test_round_robin_create_destroy (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5556
2022-12-01T10:11:42.0405184Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5557
2022-12-01T10:11:42.0405339Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5558
2022-12-01T10:11:42.0405478Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5559
2022-12-01T10:11:42.0405761Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0405887Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0406212Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0406382Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0406544Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:11:42.0406817Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0406941Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0407222Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0407346Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0407511Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:11:42.0407786Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0407911Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0408190Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0408324Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0408490Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:11:42.0408763Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0408871Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0409210Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0409347Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0409510Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:11:42.0409930Z [W ProcessGroupRoundRobin.cpp:12] Warning: ProcessGroupRoundRobin is deprecated and scheduled to be removed after this current release (1.13). Please file an issue on https://github.com/pytorch/pytorch/issues if there are any concerns or issues with this deprecation. (function ProcessGroupRoundRobin)
2022-12-01T10:11:42.0410346Z [W ProcessGroupRoundRobin.cpp:12] Warning: ProcessGroupRoundRobin is deprecated and scheduled to be removed after this current release (1.13). Please file an issue on https://github.com/pytorch/pytorch/issues if there are any concerns or issues with this deprecation. (function ProcessGroupRoundRobin)
2022-12-01T10:11:42.0410757Z [W ProcessGroupRoundRobin.cpp:12] Warning: ProcessGroupRoundRobin is deprecated and scheduled to be removed after this current release (1.13). Please file an issue on https://github.com/pytorch/pytorch/issues if there are any concerns or issues with this deprecation. (function ProcessGroupRoundRobin)
2022-12-01T10:11:42.0411164Z [W ProcessGroupRoundRobin.cpp:12] Warning: ProcessGroupRoundRobin is deprecated and scheduled to be removed after this current release (1.13). Please file an issue on https://github.com/pytorch/pytorch/issues if there are any concerns or issues with this deprecation. (function ProcessGroupRoundRobin)
2022-12-01T10:11:42.0411566Z [W ProcessGroupRoundRobin.cpp:12] Warning: ProcessGroupRoundRobin is deprecated and scheduled to be removed after this current release (1.13). Please file an issue on https://github.com/pytorch/pytorch/issues if there are any concerns or issues with this deprecation. (function ProcessGroupRoundRobin)
2022-12-01T10:11:42.0412004Z [W ProcessGroupRoundRobin.cpp:12] Warning: ProcessGroupRoundRobin is deprecated and scheduled to be removed after this current release (1.13). Please file an issue on https://github.com/pytorch/pytorch/issues if there are any concerns or issues with this deprecation. (function ProcessGroupRoundRobin)
2022-12-01T10:11:42.0412435Z [W ProcessGroupRoundRobin.cpp:12] Warning: ProcessGroupRoundRobin is deprecated and scheduled to be removed after this current release (1.13). Please file an issue on https://github.com/pytorch/pytorch/issues if there are any concerns or issues with this deprecation. (function ProcessGroupRoundRobin)
2022-12-01T10:11:42.0412835Z [W ProcessGroupRoundRobin.cpp:12] Warning: ProcessGroupRoundRobin is deprecated and scheduled to be removed after this current release (1.13). Please file an issue on https://github.com/pytorch/pytorch/issues if there are any concerns or issues with this deprecation. (function ProcessGroupRoundRobin)
2022-12-01T10:11:42.0412905Z ok (2.366s)
2022-12-01T10:11:42.0412911Z 
2022-12-01T10:11:42.0413111Z ----------------------------------------------------------------------
2022-12-01T10:11:42.0413189Z Ran 1 test in 2.367s
2022-12-01T10:11:42.0413195Z 
2022-12-01T10:11:42.0413259Z OK
2022-12-01T10:11:42.0413267Z 
2022-12-01T10:11:42.0413339Z Generating XML reports...
2022-12-01T10:11:42.0413673Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221201101056.xml
2022-12-01T10:11:42.0413907Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:11:42.0414185Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0414310Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0414594Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0414730Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0414735Z 
2022-12-01T10:11:42.0414809Z Running tests...
2022-12-01T10:11:42.0414994Z ----------------------------------------------------------------------
2022-12-01T10:11:42.0415219Z   test_scatter_basics (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5659
2022-12-01T10:11:42.0415378Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5660
2022-12-01T10:11:42.0415528Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5661
2022-12-01T10:11:42.0415678Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5662
2022-12-01T10:11:42.0415951Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0416073Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0416354Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0416489Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0416645Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:11:42.0416920Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0417040Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0417316Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0417449Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0417609Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:11:42.0417876Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0417997Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0418317Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0418546Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0418710Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:11:42.0418982Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0419103Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0419377Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0419509Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0419672Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:11:42.0419740Z ok (2.066s)
2022-12-01T10:11:42.0419745Z 
2022-12-01T10:11:42.0419930Z ----------------------------------------------------------------------
2022-12-01T10:11:42.0420007Z Ran 1 test in 2.066s
2022-12-01T10:11:42.0420015Z 
2022-12-01T10:11:42.0420079Z OK
2022-12-01T10:11:42.0420083Z 
2022-12-01T10:11:42.0420170Z Generating XML reports...
2022-12-01T10:11:42.0420645Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221201101100.xml
2022-12-01T10:11:42.0420884Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:11:42.0421159Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0421281Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0421545Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0421681Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0421687Z 
2022-12-01T10:11:42.0421765Z Running tests...
2022-12-01T10:11:42.0421962Z ----------------------------------------------------------------------
2022-12-01T10:11:42.0422192Z   test_scatter_basics_cuda (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5726
2022-12-01T10:11:42.0422347Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5727
2022-12-01T10:11:42.0422502Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5728
2022-12-01T10:11:42.0422654Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5729
2022-12-01T10:11:42.0423000Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0423120Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0423409Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0423547Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0423718Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:11:42.0423983Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0424105Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0424385Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0424517Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0424665Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:11:42.0424934Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0425122Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0425396Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0425571Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0425734Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:11:42.0426002Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0426123Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0426397Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0426516Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0426682Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:11:42.0426788Z skip: Need at least 2 CUDA devices (1.866s)
2022-12-01T10:11:42.0426796Z 
2022-12-01T10:11:42.0426992Z ----------------------------------------------------------------------
2022-12-01T10:11:42.0427068Z Ran 1 test in 1.866s
2022-12-01T10:11:42.0427074Z 
2022-12-01T10:11:42.0427146Z OK (skipped=1)
2022-12-01T10:11:42.0427150Z 
2022-12-01T10:11:42.0427235Z Generating XML reports...
2022-12-01T10:11:42.0427562Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221201101103.xml
2022-12-01T10:11:42.0427793Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:11:42.0428053Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0428175Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0428455Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0428590Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0428597Z 
2022-12-01T10:11:42.0428670Z Running tests...
2022-12-01T10:11:42.0428867Z ----------------------------------------------------------------------
2022-12-01T10:11:42.0429087Z   test_scatter_checks (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5781
2022-12-01T10:11:42.0429241Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5782
2022-12-01T10:11:42.0429379Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5783
2022-12-01T10:11:42.0429530Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5784
2022-12-01T10:11:42.0429799Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0429921Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0430206Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0430343Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0430508Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:11:42.0430777Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0430903Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0431169Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0431303Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0431461Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:11:42.0431759Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0431908Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0432185Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0432317Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0432474Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:11:42.0432730Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0432851Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0433124Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0433257Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0433416Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:11:42.0433485Z ok (2.167s)
2022-12-01T10:11:42.0433491Z 
2022-12-01T10:11:42.0433687Z ----------------------------------------------------------------------
2022-12-01T10:11:42.0433764Z Ran 1 test in 2.167s
2022-12-01T10:11:42.0433768Z 
2022-12-01T10:11:42.0433816Z OK
2022-12-01T10:11:42.0433834Z 
2022-12-01T10:11:42.0433906Z Generating XML reports...
2022-12-01T10:11:42.0434234Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221201101107.xml
2022-12-01T10:11:42.0434464Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:11:42.0434735Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0434858Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0435143Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0435282Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0435287Z 
2022-12-01T10:11:42.0435361Z Running tests...
2022-12-01T10:11:42.0435541Z ----------------------------------------------------------------------
2022-12-01T10:11:42.0435762Z   test_scatter_stress (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5848
2022-12-01T10:11:42.0435917Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5849
2022-12-01T10:11:42.0436070Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5850
2022-12-01T10:11:42.0436238Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5851
2022-12-01T10:11:42.0436511Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0436634Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0436913Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0437047Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0437199Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:11:42.0437470Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0437639Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0437947Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0438082Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0438279Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:11:42.0438575Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0438697Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0438959Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0439094Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0439261Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:11:42.0439536Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0439659Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0439939Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0440073Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0440239Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:11:42.0440294Z ok (3.767s)
2022-12-01T10:11:42.0440312Z 
2022-12-01T10:11:42.0440495Z ----------------------------------------------------------------------
2022-12-01T10:11:42.0440571Z Ran 1 test in 3.767s
2022-12-01T10:11:42.0440577Z 
2022-12-01T10:11:42.0440638Z OK
2022-12-01T10:11:42.0440643Z 
2022-12-01T10:11:42.0440729Z Generating XML reports...
2022-12-01T10:11:42.0441057Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221201101110.xml
2022-12-01T10:11:42.0441286Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:11:42.0441561Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0441685Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0441953Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0442090Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0442095Z 
2022-12-01T10:11:42.0442167Z Running tests...
2022-12-01T10:11:42.0442364Z ----------------------------------------------------------------------
2022-12-01T10:11:42.0442592Z   test_scatter_stress_cuda (__main__.ProcessGroupGlooTest) ... skip: Test is flaky, see https://github.com/pytorch/pytorch/issues/15963 (0.001s)
2022-12-01T10:11:42.0442597Z 
2022-12-01T10:11:42.0442790Z ----------------------------------------------------------------------
2022-12-01T10:11:42.0442865Z Ran 1 test in 0.001s
2022-12-01T10:11:42.0442871Z 
2022-12-01T10:11:42.0442942Z OK (skipped=1)
2022-12-01T10:11:42.0442947Z 
2022-12-01T10:11:42.0443032Z Generating XML reports...
2022-12-01T10:11:42.0443343Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221201101116.xml
2022-12-01T10:11:42.0443578Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:11:42.0443860Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0443982Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0444262Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0444396Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0444401Z 
2022-12-01T10:11:42.0444474Z Running tests...
2022-12-01T10:11:42.0444667Z ----------------------------------------------------------------------
2022-12-01T10:11:42.0444911Z   test_send_recv_all_to_all (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5949
2022-12-01T10:11:42.0445096Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5950
2022-12-01T10:11:42.0445251Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5951
2022-12-01T10:11:42.0445404Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5952
2022-12-01T10:11:42.0445680Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0445803Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0446085Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0446222Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0446390Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:11:42.0446647Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0446774Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0447053Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0447188Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0447353Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:11:42.0447622Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0447747Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0448025Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0448161Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0448314Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:11:42.0448581Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0448704Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0448979Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0449111Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0449271Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:11:42.0449340Z ok (1.969s)
2022-12-01T10:11:42.0449345Z 
2022-12-01T10:11:42.0449539Z ----------------------------------------------------------------------
2022-12-01T10:11:42.0449601Z Ran 1 test in 1.969s
2022-12-01T10:11:42.0449608Z 
2022-12-01T10:11:42.0449669Z OK
2022-12-01T10:11:42.0449676Z 
2022-12-01T10:11:42.0449759Z Generating XML reports...
2022-12-01T10:11:42.0450083Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221201101117.xml
2022-12-01T10:11:42.0450316Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:11:42.0450590Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0450714Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0450994Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0451118Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0451135Z 
2022-12-01T10:11:42.0451195Z Running tests...
2022-12-01T10:11:42.0451430Z ----------------------------------------------------------------------
2022-12-01T10:11:42.0451654Z   test_sparse_allreduce_basics (__main__.ProcessGroupGlooTest) ... skip: intermittent failures on Windows, in CI (0.000s)
2022-12-01T10:11:42.0451659Z 
2022-12-01T10:11:42.0451854Z ----------------------------------------------------------------------
2022-12-01T10:11:42.0451931Z Ran 1 test in 0.000s
2022-12-01T10:11:42.0451936Z 
2022-12-01T10:11:42.0452007Z OK (skipped=1)
2022-12-01T10:11:42.0452011Z 
2022-12-01T10:11:42.0452094Z Generating XML reports...
2022-12-01T10:11:42.0452414Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221201101121.xml
2022-12-01T10:11:42.0452630Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:11:42.0452903Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0453026Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0453304Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0453442Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0453447Z 
2022-12-01T10:11:42.0453518Z Running tests...
2022-12-01T10:11:42.0453715Z ----------------------------------------------------------------------
2022-12-01T10:11:42.0453954Z   test_sparse_allreduce_basics_cuda (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6026
2022-12-01T10:11:42.0454111Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6027
2022-12-01T10:11:42.0454248Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6028
2022-12-01T10:11:42.0454399Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 6029
2022-12-01T10:11:42.0454672Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0454797Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0455074Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0455207Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0455369Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:11:42.0455642Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0455751Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0456031Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0456168Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0456334Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:11:42.0456604Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0456729Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0457004Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0457141Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0457303Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:11:42.0457561Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0457684Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0457990Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0458157Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0458316Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:11:42.0458488Z skip: Need at least 2 CUDA devices (2.070s)
2022-12-01T10:11:42.0458494Z 
2022-12-01T10:11:42.0458695Z ----------------------------------------------------------------------
2022-12-01T10:11:42.0458773Z Ran 1 test in 2.070s
2022-12-01T10:11:42.0458777Z 
2022-12-01T10:11:42.0458835Z OK (skipped=1)
2022-12-01T10:11:42.0458858Z 
2022-12-01T10:11:42.0458929Z Generating XML reports...
2022-12-01T10:11:42.0459250Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221201101122.xml
2022-12-01T10:11:42.0459480Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:11:42.0459757Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0459884Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0460163Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0460300Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0460304Z 
2022-12-01T10:11:42.0460480Z Running tests...
2022-12-01T10:11:42.0460662Z ----------------------------------------------------------------------
2022-12-01T10:11:42.0460897Z   test_sparse_allreduce_checks (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6081
2022-12-01T10:11:42.0461051Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6082
2022-12-01T10:11:42.0461204Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6083
2022-12-01T10:11:42.0461359Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 6084
2022-12-01T10:11:42.0461637Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0461760Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0462040Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0462176Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0462326Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:11:42.0462593Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0462715Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0462996Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0463131Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0463295Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:11:42.0463561Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0463683Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0463944Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0464078Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0464238Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:11:42.0464576Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0464702Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0465016Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0465164Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0465324Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:11:42.0465381Z ok (2.066s)
2022-12-01T10:11:42.0465399Z 
2022-12-01T10:11:42.0465581Z ----------------------------------------------------------------------
2022-12-01T10:11:42.0465656Z Ran 1 test in 2.066s
2022-12-01T10:11:42.0465661Z 
2022-12-01T10:11:42.0465725Z OK
2022-12-01T10:11:42.0465729Z 
2022-12-01T10:11:42.0465814Z Generating XML reports...
2022-12-01T10:11:42.0466140Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221201101126.xml
2022-12-01T10:11:42.0466373Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:11:42.0466647Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0466770Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0467035Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0467170Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0467174Z 
2022-12-01T10:11:42.0467248Z Running tests...
2022-12-01T10:11:42.0467442Z ----------------------------------------------------------------------
2022-12-01T10:11:42.0467566Z   test_forward_backward (__main__.ReducerTest) ... ok (0.006s)
2022-12-01T10:11:42.0467572Z 
2022-12-01T10:11:42.0467759Z ----------------------------------------------------------------------
2022-12-01T10:11:42.0467834Z Ran 1 test in 0.013s
2022-12-01T10:11:42.0467842Z 
2022-12-01T10:11:42.0467903Z OK
2022-12-01T10:11:42.0467910Z 
2022-12-01T10:11:42.0467994Z Generating XML reports...
2022-12-01T10:11:42.0468274Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ReducerTest-20221201101130.xml
2022-12-01T10:11:42.0468502Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:11:42.0468771Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0468894Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0469173Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0469308Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0469313Z 
2022-12-01T10:11:42.0469385Z Running tests...
2022-12-01T10:11:42.0469578Z ----------------------------------------------------------------------
2022-12-01T10:11:42.0470227Z   test_forward_backward_optimizer (__main__.ReducerTest) ... [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-12-01T10:11:42.0470296Z ok (0.008s)
2022-12-01T10:11:42.0470302Z 
2022-12-01T10:11:42.0470479Z ----------------------------------------------------------------------
2022-12-01T10:11:42.0470553Z Ran 1 test in 0.013s
2022-12-01T10:11:42.0470558Z 
2022-12-01T10:11:42.0470619Z OK
2022-12-01T10:11:42.0470624Z 
2022-12-01T10:11:42.0470740Z Generating XML reports...
2022-12-01T10:11:42.0471037Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ReducerTest-20221201101131.xml
2022-12-01T10:11:42.0471295Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:11:42.0471566Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0471690Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0471961Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0472095Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0472101Z 
2022-12-01T10:11:42.0472174Z Running tests...
2022-12-01T10:11:42.0472371Z ----------------------------------------------------------------------
2022-12-01T10:11:42.0472521Z   test_forward_backward_unused_parameters (__main__.ReducerTest) ... ok (0.006s)
2022-12-01T10:11:42.0472526Z 
2022-12-01T10:11:42.0472720Z ----------------------------------------------------------------------
2022-12-01T10:11:42.0472799Z Ran 1 test in 0.013s
2022-12-01T10:11:42.0472804Z 
2022-12-01T10:11:42.0472865Z OK
2022-12-01T10:11:42.0472871Z 
2022-12-01T10:11:42.0472955Z Generating XML reports...
2022-12-01T10:11:42.0473238Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ReducerTest-20221201101133.xml
2022-12-01T10:11:42.0473467Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:11:42.0473740Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0473864Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0474144Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0474278Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0474285Z 
2022-12-01T10:11:42.0474358Z Running tests...
2022-12-01T10:11:42.0474550Z ----------------------------------------------------------------------
2022-12-01T10:11:42.0474667Z   test_multi_dtype_multi_bucket (__main__.ReducerTest) ... ok (0.004s)
2022-12-01T10:11:42.0474685Z 
2022-12-01T10:11:42.0474863Z ----------------------------------------------------------------------
2022-12-01T10:11:42.0474939Z Ran 1 test in 0.013s
2022-12-01T10:11:42.0474945Z 
2022-12-01T10:11:42.0475005Z OK
2022-12-01T10:11:42.0475009Z 
2022-12-01T10:11:42.0475092Z Generating XML reports...
2022-12-01T10:11:42.0475385Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ReducerTest-20221201101134.xml
2022-12-01T10:11:42.0475615Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:11:42.0475890Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0476015Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0476282Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0476418Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0476423Z 
2022-12-01T10:11:42.0476494Z Running tests...
2022-12-01T10:11:42.0476687Z ----------------------------------------------------------------------
2022-12-01T10:11:42.0476822Z   test_multi_dtype_single_bucket (__main__.ReducerTest) ... ok (0.007s)
2022-12-01T10:11:42.0476826Z 
2022-12-01T10:11:42.0477017Z ----------------------------------------------------------------------
2022-12-01T10:11:42.0477092Z Ran 1 test in 0.013s
2022-12-01T10:11:42.0477097Z 
2022-12-01T10:11:42.0477158Z OK
2022-12-01T10:11:42.0477163Z 
2022-12-01T10:11:42.0477246Z Generating XML reports...
2022-12-01T10:11:42.0477557Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ReducerTest-20221201101136.xml
2022-12-01T10:11:42.0477812Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:11:42.0478086Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0478207Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0478488Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0478624Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0478629Z 
2022-12-01T10:11:42.0478701Z Running tests...
2022-12-01T10:11:42.0478897Z ----------------------------------------------------------------------
2022-12-01T10:11:42.0479017Z   test_single_dtype_single_bucket (__main__.ReducerTest) ... ok (0.004s)
2022-12-01T10:11:42.0479024Z 
2022-12-01T10:11:42.0479216Z ----------------------------------------------------------------------
2022-12-01T10:11:42.0479294Z Ran 1 test in 0.013s
2022-12-01T10:11:42.0479299Z 
2022-12-01T10:11:42.0479363Z OK
2022-12-01T10:11:42.0479368Z 
2022-12-01T10:11:42.0479451Z Generating XML reports...
2022-12-01T10:11:42.0479744Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ReducerTest-20221201101137.xml
2022-12-01T10:11:42.0479975Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:11:42.0480248Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0480371Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0480637Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0480776Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0480780Z 
2022-12-01T10:11:42.0480857Z Running tests...
2022-12-01T10:11:42.0481052Z ----------------------------------------------------------------------
2022-12-01T10:11:42.0481281Z   test_logging_init (__main__.RendezvousEnvTest) ... INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:11:42.0481582Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-12-01T10:11:42.0481650Z ok (0.258s)
2022-12-01T10:11:42.0481655Z 
2022-12-01T10:11:42.0481844Z ----------------------------------------------------------------------
2022-12-01T10:11:42.0481906Z Ran 1 test in 0.258s
2022-12-01T10:11:42.0481924Z 
2022-12-01T10:11:42.0481973Z OK
2022-12-01T10:11:42.0481977Z 
2022-12-01T10:11:42.0482059Z Generating XML reports...
2022-12-01T10:11:42.0482377Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-RendezvousEnvTest-20221201101139.xml
2022-12-01T10:11:42.0482608Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:11:42.0482883Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:11:42.0483006Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:11:42.0483284Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:11:42.0483417Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:11:42.0483422Z 
2022-12-01T10:11:42.0483481Z Running tests...
2022-12-01T10:11:42.0483673Z ----------------------------------------------------------------------
2022-12-01T10:11:42.0484365Z   test_default_store_timeout_gloo (__main__.TimeoutTest) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/74714 for allplatform(s) . If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.245s)
2022-12-01T10:11:42.0484395Z 
2022-12-01T10:11:42.0484588Z ----------------------------------------------------------------------
2022-12-01T10:11:42.0484663Z Ran 1 test in 0.245s
2022-12-01T10:11:42.0484668Z 
2022-12-01T10:11:42.0484739Z OK (skipped=1)
2022-12-01T10:11:42.0484744Z 
2022-12-01T10:11:42.0484830Z Generating XML reports...
2022-12-01T10:11:42.0485126Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-TimeoutTest-20221201101141.xml
2022-12-01T10:11:42.0485131Z 
2022-12-01T10:11:42.0485476Z ##[endgroup]
2022-12-01T10:11:42.0485789Z FINISHED PRINTING LOG FILE of distributed/test_c10d_gloo (/var/lib/jenkins/workspace/test/test-reports/distributed-test_c10d_gloo_zqx1eqdr)
2022-12-01T10:11:42.0485808Z 
2022-12-01T10:11:42.0485999Z Running distributed/fsdp/test_fsdp_state_dict ... [2022-12-01 10:11:41.937653]
2022-12-01T10:11:42.0486373Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_state_dict.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-12-01 10:11:41.937982]
2022-12-01T10:13:14.5931191Z 
2022-12-01T10:13:14.5931767Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_state_dict
2022-12-01T10:13:14.5932881Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_state_dict (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_state_dict_3vynux9v)
2022-12-01T10:13:14.5933755Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_state_dict
2022-12-01T10:13:14.5934058Z 
2022-12-01T10:13:14.5934171Z Running tests...
2022-12-01T10:13:14.5934626Z ----------------------------------------------------------------------
2022-12-01T10:13:14.5935280Z   test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_False_state_dict_rank0_and_offload_False (__main__.TestFSDPStateDict)
2022-12-01T10:13:14.5977454Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6260
2022-12-01T10:13:14.5978251Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6261
2022-12-01T10:13:14.5979101Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:13:14.5979589Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:13:14.5980225Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:13:14.5980895Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:13:14.5981398Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:13:14.5982091Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:13:14.5982536Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:13:14.5983172Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:13:14.5983711Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:13:14.5984199Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:13:14.5984915Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:13:14.5987628Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:13:14.5988213Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:13:14.5988920Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:13:14.5989375Z dist init r=0, world=2
2022-12-01T10:13:14.5989637Z dist init r=1, world=2
2022-12-01T10:13:14.5989964Z skip: Need at least 2 CUDA devices (1.450s)
2022-12-01T10:13:14.5991302Z   test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_False_state_dict_rank0_and_offload_True (__main__.TestFSDPStateDict)
2022-12-01T10:13:14.5991808Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6288
2022-12-01T10:13:14.5992202Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6289
2022-12-01T10:13:14.5992718Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:13:14.5993039Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:13:14.5993496Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:13:14.5993850Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:13:14.5994192Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:13:14.5994649Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:13:14.5994974Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:13:14.5995397Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:13:14.5995726Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:13:14.5996063Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:13:14.5996564Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:13:14.5997084Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:13:14.5997457Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:13:14.5997805Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:13:14.5998062Z dist init r=1, world=2
2022-12-01T10:13:14.5998246Z dist init r=0, world=2
2022-12-01T10:13:14.5998441Z skip: Need at least 2 CUDA devices (1.311s)
2022-12-01T10:13:14.6041203Z   test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_True_state_dict_rank0_and_offload_False (__main__.TestFSDPStateDict)
2022-12-01T10:13:14.6041760Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6316
2022-12-01T10:13:14.6042154Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6317
2022-12-01T10:13:14.6042693Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:13:14.6043018Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:13:14.6043453Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:13:14.6043805Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:13:14.6044143Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:13:14.6044605Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:13:14.6044937Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:13:14.6045535Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:13:14.6045946Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:13:14.6046269Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:13:14.6046772Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:13:14.6047294Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:13:14.6047671Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:13:14.6048021Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:13:14.6048280Z dist init r=0, world=2
2022-12-01T10:13:14.6048466Z dist init r=1, world=2
2022-12-01T10:13:14.6048664Z skip: Need at least 2 CUDA devices (1.311s)
2022-12-01T10:13:14.6049059Z   test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_True_state_dict_rank0_and_offload_True (__main__.TestFSDPStateDict)
2022-12-01T10:13:14.6049548Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6344
2022-12-01T10:13:14.6049937Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6345
2022-12-01T10:13:14.6050376Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:13:14.6050705Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:13:14.6051136Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:13:14.6051478Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:13:14.6051812Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:13:14.6052275Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:13:14.6052602Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:13:14.6053009Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:13:14.6053345Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:13:14.6053679Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:13:14.6054157Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:13:14.6054678Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:13:14.6055069Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:13:14.6055410Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:13:14.6055660Z dist init r=1, world=2
2022-12-01T10:13:14.6055840Z dist init r=0, world=2
2022-12-01T10:13:14.6056052Z skip: Need at least 2 CUDA devices (1.212s)
2022-12-01T10:13:14.6056430Z   test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_False_state_dict_rank0_and_offload_False (__main__.TestFSDPStateDict)
2022-12-01T10:13:14.6056928Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6372
2022-12-01T10:13:14.6057316Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6373
2022-12-01T10:13:14.6057805Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:13:14.6058236Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:13:14.6058674Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:13:14.6059028Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:13:14.6059365Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:13:14.6059823Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:13:14.6060152Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:13:14.6060747Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:13:14.6061085Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:13:14.6061421Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:13:14.6061918Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:13:14.6062442Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:13:14.6062817Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:13:14.6063163Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:13:14.6063423Z dist init r=1, world=2
2022-12-01T10:13:14.6063599Z dist init r=0, world=2
2022-12-01T10:13:14.6063796Z skip: Need at least 2 CUDA devices (1.312s)
2022-12-01T10:13:14.6064189Z   test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_False_state_dict_rank0_and_offload_True (__main__.TestFSDPStateDict)
2022-12-01T10:13:14.6064684Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6400
2022-12-01T10:13:14.6065058Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6401
2022-12-01T10:13:14.6065509Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:13:14.6065836Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:13:14.6066260Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:13:14.6066590Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:13:14.6066924Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:13:14.6067395Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:13:14.6067730Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:13:14.6068139Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:13:14.6068483Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:13:14.6068818Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:13:14.6069303Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:13:14.6069822Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:13:14.6070207Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:13:14.6070625Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:13:14.6070955Z dist init r=1, world=2
2022-12-01T10:13:14.6071137Z dist init r=0, world=2
2022-12-01T10:13:14.6071351Z skip: Need at least 2 CUDA devices (1.312s)
2022-12-01T10:13:14.6071732Z   test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_True_state_dict_rank0_and_offload_False (__main__.TestFSDPStateDict)
2022-12-01T10:13:14.6072225Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6428
2022-12-01T10:13:14.6072614Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6429
2022-12-01T10:13:14.6073063Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:13:14.6073380Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:13:14.6073812Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:13:14.6074157Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:13:14.6074495Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:13:14.6074948Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:13:14.6075278Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:13:14.6075702Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:13:14.6076030Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:13:14.6076365Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:13:14.6076866Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:13:14.6077384Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:13:14.6077757Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:13:14.6078104Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:13:14.6078364Z dist init r=1, world=2
2022-12-01T10:13:14.6078533Z dist init r=0, world=2
2022-12-01T10:13:14.6078742Z skip: Need at least 2 CUDA devices (1.312s)
2022-12-01T10:13:14.6079135Z   test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_True_state_dict_rank0_and_offload_True (__main__.TestFSDPStateDict)
2022-12-01T10:13:14.6079627Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6456
2022-12-01T10:13:14.6080004Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6457
2022-12-01T10:13:14.6080453Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:13:14.6080783Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:13:14.6081212Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:13:14.6081547Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:13:14.6081884Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:13:14.6082356Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:13:14.6082671Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:13:14.6083140Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:13:14.6083519Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:13:14.6083853Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:13:14.6084330Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:13:14.6084844Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:13:14.6085230Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:13:14.6085580Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:13:14.6085833Z dist init r=1, world=2
2022-12-01T10:13:14.6086018Z dist init r=0, world=2
2022-12-01T10:13:14.6086232Z skip: Need at least 2 CUDA devices (1.311s)
2022-12-01T10:13:14.6086621Z   test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_False_state_dict_rank0_and_offload_False (__main__.TestFSDPStateDict)
2022-12-01T10:13:14.6087118Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6484
2022-12-01T10:13:14.6087507Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6485
2022-12-01T10:13:14.6087959Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:13:14.6088278Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:13:14.6088706Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:13:14.6089054Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:13:14.6089376Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:13:14.6089848Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:13:14.6090175Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:13:14.6090606Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:13:14.6090933Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:13:14.6091267Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:13:14.6091763Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:13:14.6092283Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:13:14.6092657Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:13:14.6093008Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:13:14.6093266Z dist init r=1, world=2
2022-12-01T10:13:14.6093435Z dist init r=0, world=2
2022-12-01T10:13:14.6093648Z skip: Need at least 2 CUDA devices (1.311s)
2022-12-01T10:13:14.6094042Z   test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_False_state_dict_rank0_and_offload_True (__main__.TestFSDPStateDict)
2022-12-01T10:13:14.6094541Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6512
2022-12-01T10:13:14.6094915Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6513
2022-12-01T10:13:14.6095406Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:13:14.6095766Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:13:14.6096178Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:13:14.6096524Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:13:14.6096873Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:13:14.6097346Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:13:14.6097664Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:13:14.6098090Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:13:14.6098524Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:13:14.6098866Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:13:14.6099355Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:13:14.6099879Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:13:14.6100268Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:13:14.6100751Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:13:14.6101013Z dist init r=1, world=2
2022-12-01T10:13:14.6101196Z dist init r=0, world=2
2022-12-01T10:13:14.6101409Z skip: Need at least 2 CUDA devices (1.312s)
2022-12-01T10:13:14.6101793Z   test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_True_state_dict_rank0_and_offload_False (__main__.TestFSDPStateDict)
2022-12-01T10:13:14.6102293Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6540
2022-12-01T10:13:14.6102681Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6541
2022-12-01T10:13:14.6103135Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:13:14.6103451Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:13:14.6103886Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:13:14.6104236Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:13:14.6104559Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:13:14.6105028Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:13:14.6105362Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:13:14.6105792Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:13:14.6106124Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:13:14.6106461Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:13:14.6106954Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:13:14.6107472Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:13:14.6107845Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:13:14.6108267Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:13:14.6108571Z dist init r=1, world=2
2022-12-01T10:13:14.6108739Z dist init r=0, world=2
2022-12-01T10:13:14.6108950Z skip: Need at least 2 CUDA devices (1.312s)
2022-12-01T10:13:14.6109346Z   test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_True_state_dict_rank0_and_offload_True (__main__.TestFSDPStateDict)
2022-12-01T10:13:14.6109845Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6568
2022-12-01T10:13:14.6110221Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6569
2022-12-01T10:13:14.6110677Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:13:14.6111009Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:13:14.6111426Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:13:14.6111773Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:13:14.6112109Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:13:14.6112578Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:13:14.6112892Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:13:14.6113322Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:13:14.6113667Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:13:14.6113999Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:13:14.6114483Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:13:14.6115001Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:13:14.6115389Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:13:14.6115720Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:13:14.6115983Z dist init r=0, world=2
2022-12-01T10:13:14.6116164Z dist init r=1, world=2
2022-12-01T10:13:14.6116358Z skip: Need at least 2 CUDA devices (1.312s)
2022-12-01T10:13:14.6116756Z   test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_False_state_dict_rank0_and_offload_False (__main__.TestFSDPStateDict)
2022-12-01T10:13:14.6117256Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6596
2022-12-01T10:13:14.6117647Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6597
2022-12-01T10:13:14.6118084Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:13:14.6118416Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:13:14.6118848Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:13:14.6119198Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:13:14.6119521Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:13:14.6119996Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:13:14.6120365Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:13:14.6120795Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:13:14.6121165Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:13:14.6121504Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:13:14.6121998Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:13:14.6122502Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:13:14.6122888Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:13:14.6123238Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:13:14.6123497Z dist init r=1, world=2
2022-12-01T10:13:14.6123667Z dist init r=0, world=2
2022-12-01T10:13:14.6123875Z skip: Need at least 2 CUDA devices (1.311s)
2022-12-01T10:13:14.6124274Z   test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_False_state_dict_rank0_and_offload_True (__main__.TestFSDPStateDict)
2022-12-01T10:13:14.6124755Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6624
2022-12-01T10:13:14.6125144Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6625
2022-12-01T10:13:14.6125598Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:13:14.6125926Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:13:14.6126344Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:13:14.6126696Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:13:14.6127036Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:13:14.6127509Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:13:14.6127821Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:13:14.6128252Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:13:14.6128596Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:13:14.6128920Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:13:14.6129418Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:13:14.6129935Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:13:14.6130324Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:13:14.6130659Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:13:14.6130914Z dist init r=0, world=2
2022-12-01T10:13:14.6131096Z dist init r=1, world=2
2022-12-01T10:13:14.6131292Z skip: Need at least 2 CUDA devices (1.312s)
2022-12-01T10:13:14.6131690Z   test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_True_state_dict_rank0_and_offload_False (__main__.TestFSDPStateDict)
2022-12-01T10:13:14.6132181Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6652
2022-12-01T10:13:14.6132567Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6653
2022-12-01T10:13:14.6133041Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:13:14.6133405Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:13:14.6133834Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:13:14.6134183Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:13:14.6134508Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:13:14.6134980Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:13:14.6135311Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:13:14.6135725Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:13:14.6136070Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:13:14.6136408Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:13:14.6136902Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:13:14.6137405Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:13:14.6137789Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:13:14.6138236Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:13:14.6138500Z dist init r=1, world=2
2022-12-01T10:13:14.6138722Z dist init r=0, world=2
2022-12-01T10:13:14.6138932Z skip: Need at least 2 CUDA devices (1.312s)
2022-12-01T10:13:14.6139329Z   test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_True_state_dict_rank0_and_offload_True (__main__.TestFSDPStateDict)
2022-12-01T10:13:14.6139817Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6680
2022-12-01T10:13:14.6140207Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6681
2022-12-01T10:13:14.6140802Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:13:14.6141142Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:13:14.6141558Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:13:14.6141907Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:13:14.6142248Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:13:14.6142721Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:13:14.6143040Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:13:14.6143469Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:13:14.6143820Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:13:14.6144142Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:13:14.6144636Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:13:14.6145157Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:13:14.6145616Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:13:14.6145955Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:13:14.6146261Z dist init r=1, world=2
2022-12-01T10:13:14.6146445Z dist init r=0, world=2
2022-12-01T10:13:14.6146641Z skip: Need at least 2 CUDA devices (1.312s)
2022-12-01T10:13:14.6147033Z   test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_False_state_dict_rank0_and_offload_False (__main__.TestFSDPStateDict)
2022-12-01T10:13:14.6147521Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6708
2022-12-01T10:13:14.6147909Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6709
2022-12-01T10:13:14.6148353Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:13:14.6148686Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:13:14.6149118Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:13:14.6149469Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:13:14.6149797Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:13:14.6150266Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:13:14.6150595Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:13:14.6151012Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:13:14.6151355Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:13:14.6151691Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:13:14.6152188Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:13:14.6152697Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:13:14.6153087Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:13:14.6153433Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:13:14.6153697Z dist init r=1, world=2
2022-12-01T10:13:14.6153867Z dist init r=0, world=2
2022-12-01T10:13:14.6154074Z skip: Need at least 2 CUDA devices (1.312s)
2022-12-01T10:13:14.6154464Z   test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_False_state_dict_rank0_and_offload_True (__main__.TestFSDPStateDict)
2022-12-01T10:13:14.6154946Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6736
2022-12-01T10:13:14.6155337Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6737
2022-12-01T10:13:14.6155791Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:13:14.6156120Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:13:14.6156535Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:13:14.6156878Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:13:14.6157216Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:13:14.6157669Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:13:14.6158040Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:13:14.6158469Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:13:14.6158864Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:13:14.6159191Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:13:14.6159688Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:13:14.6160212Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:13:14.6160601Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:13:14.6160934Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:13:14.6161193Z dist init r=0, world=2
2022-12-01T10:13:14.6161375Z dist init r=1, world=2
2022-12-01T10:13:14.6161572Z skip: Need at least 2 CUDA devices (1.312s)
2022-12-01T10:13:14.6161961Z   test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_True_state_dict_rank0_and_offload_False (__main__.TestFSDPStateDict)
2022-12-01T10:13:14.6162457Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6764
2022-12-01T10:13:14.6162844Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6765
2022-12-01T10:13:14.6163302Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:13:14.6163635Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:13:14.6164063Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:13:14.6164401Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:13:14.6164742Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:13:14.6165216Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:13:14.6165543Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:13:14.6165959Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:13:14.6166307Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:13:14.6166645Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:13:14.6167141Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:13:14.6167647Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:13:14.6168041Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:13:14.6168386Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:13:14.6168634Z dist init r=1, world=2
2022-12-01T10:13:14.6168814Z dist init r=0, world=2
2022-12-01T10:13:14.6169026Z skip: Need at least 2 CUDA devices (1.312s)
2022-12-01T10:13:14.6169413Z   test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_True_state_dict_rank0_and_offload_True (__main__.TestFSDPStateDict)
2022-12-01T10:13:14.6169919Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6792
2022-12-01T10:13:14.6170308Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6793
2022-12-01T10:13:14.6170818Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:13:14.6171172Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:13:14.6171603Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:13:14.6171949Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:13:14.6172287Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:13:14.6172740Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:13:14.6173067Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:13:14.6173496Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:13:14.6173845Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:13:14.6174170Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:13:14.6174668Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:13:14.6175192Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:13:14.6175565Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:13:14.6175916Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:13:14.6176167Z dist init r=1, world=2
2022-12-01T10:13:14.6176349Z dist init r=0, world=2
2022-12-01T10:13:14.6176548Z skip: Need at least 2 CUDA devices (1.211s)
2022-12-01T10:13:14.6176948Z   test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_False_state_dict_rank0_and_offload_False (__main__.TestFSDPStateDict)
2022-12-01T10:13:14.6177443Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6820
2022-12-01T10:13:14.6177833Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6821
2022-12-01T10:13:14.6178366Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:13:14.6178703Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:13:14.6179143Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:13:14.6179475Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:13:14.6179813Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:13:14.6180287Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:13:14.6180833Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:13:14.6181258Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:13:14.6181603Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:13:14.6181936Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:13:14.6182420Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:13:14.6182944Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:13:14.6183334Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:13:14.6183768Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:13:14.6184059Z dist init r=1, world=2
2022-12-01T10:13:14.6184248Z dist init r=0, world=2
2022-12-01T10:13:14.6184460Z skip: Need at least 2 CUDA devices (1.312s)
2022-12-01T10:13:14.6184839Z   test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_False_state_dict_rank0_and_offload_True (__main__.TestFSDPStateDict)
2022-12-01T10:13:14.6185326Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6848
2022-12-01T10:13:14.6185717Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6849
2022-12-01T10:13:14.6186172Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:13:14.6186494Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:13:14.6186920Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:13:14.6187267Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:13:14.6187608Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:13:14.6188062Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:13:14.6188390Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:13:14.6188819Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:13:14.6189147Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:13:14.6189483Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:13:14.6189978Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:13:14.6190495Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:13:14.6190866Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:13:14.6191218Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:13:14.6191475Z dist init r=1, world=2
2022-12-01T10:13:14.6191669Z dist init r=0, world=2
2022-12-01T10:13:14.6191866Z skip: Need at least 2 CUDA devices (1.312s)
2022-12-01T10:13:14.6192251Z   test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_True_state_dict_rank0_and_offload_False (__main__.TestFSDPStateDict)
2022-12-01T10:13:14.6192735Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6876
2022-12-01T10:13:14.6193113Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6877
2022-12-01T10:13:14.6193564Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:13:14.6193893Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:13:14.6194326Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:13:14.6194655Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:13:14.6194993Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:13:14.6195462Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:13:14.6195790Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:13:14.6196252Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:13:14.6196634Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:13:14.6196971Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:13:14.6197450Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:13:14.6197965Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:13:14.6198357Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:13:14.6198707Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:13:14.6198952Z dist init r=1, world=2
2022-12-01T10:13:14.6199136Z dist init r=0, world=2
2022-12-01T10:13:14.6199346Z skip: Need at least 2 CUDA devices (1.311s)
2022-12-01T10:13:14.6199723Z   test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_True_state_dict_rank0_and_offload_True (__main__.TestFSDPStateDict)
2022-12-01T10:13:14.6200206Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6904
2022-12-01T10:13:14.6200594Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6905
2022-12-01T10:13:14.6201048Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:13:14.6201368Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:13:14.6201798Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:13:14.6202147Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:13:14.6202487Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:13:14.6202940Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:13:14.6203269Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:13:14.6203693Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:13:14.6204023Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:13:14.6204362Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:13:14.6204858Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:13:14.6205374Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:13:14.6205747Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:13:14.6206100Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:13:14.6206356Z dist init r=1, world=2
2022-12-01T10:13:14.6206522Z dist init r=0, world=2
2022-12-01T10:13:14.6206733Z skip: Need at least 2 CUDA devices (1.211s)
2022-12-01T10:13:14.6207114Z   test_fsdp_state_dict_keys_state_dict_type_local_state_dict (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6932
2022-12-01T10:13:14.6207527Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6933
2022-12-01T10:13:14.6207964Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:13:14.6208334Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:13:14.6208770Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:13:14.6209139Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:13:14.6209477Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:13:14.6209945Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:13:14.6210271Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:13:14.6210689Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:13:14.6211032Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:13:14.6211369Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:13:14.6211865Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:13:14.6212371Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:13:14.6212757Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:13:14.6213104Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:13:14.6213350Z dist init r=1, world=2
2022-12-01T10:13:14.6213530Z dist init r=0, world=2
2022-12-01T10:13:14.6213740Z skip: Need at least 2 CUDA devices (1.311s)
2022-12-01T10:13:14.6214126Z   test_fsdp_state_dict_keys_state_dict_type_sharded_state_dict (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6960
2022-12-01T10:13:14.6214530Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6961
2022-12-01T10:13:14.6214982Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:13:14.6215317Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:13:14.6215736Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:13:14.6216082Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:13:14.6216420Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:13:14.6216886Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:13:14.6217201Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:13:14.6217630Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:13:14.6217974Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:13:14.6218410Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:13:14.6218897Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:13:14.6219418Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:13:14.6219804Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:13:14.6220138Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:13:14.6220545Z dist init r=1, world=2
2022-12-01T10:13:14.6220731Z dist init r=0, world=2
2022-12-01T10:13:14.6220942Z skip: Need at least 2 CUDA devices (1.310s)
2022-12-01T10:13:14.6221403Z   test_fsdp_state_dict_keys_state_dict_type_state_dict (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6988
2022-12-01T10:13:14.6221850Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6989
2022-12-01T10:13:14.6222309Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:13:14.6222667Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:13:14.6223096Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:13:14.6223441Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:13:14.6223778Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:13:14.6224232Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:13:14.6224559Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:13:14.6224987Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:13:14.6225321Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:13:14.6225661Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:13:14.6226157Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:13:14.6226677Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:13:14.6227050Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:13:14.6227399Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:13:14.6227653Z dist init r=1, world=2
2022-12-01T10:13:14.6227835Z dist init r=0, world=2
2022-12-01T10:13:14.6228032Z skip: Need at least 2 CUDA devices (1.210s)
2022-12-01T10:13:14.6228380Z   test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_sharded_state_dict_checkpoint_wrap_both (__main__.TestFSDPStateDict)
2022-12-01T10:13:14.6228950Z Tests saving the state dict, zeroing a target model's parameters, and ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7016
2022-12-01T10:13:14.6229334Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7017
2022-12-01T10:13:14.6229783Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:13:14.6230117Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:13:14.6230545Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:13:14.6230884Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:13:14.6231222Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:13:14.6231692Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:13:14.6232007Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:13:14.6232432Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:13:14.6232774Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:13:14.6233109Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:13:14.6233590Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:13:14.6234176Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:13:14.6234590Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:13:14.6234931Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:13:14.6235174Z dist init r=1, world=2
2022-12-01T10:13:14.6235356Z dist init r=0, world=2
2022-12-01T10:13:14.6235564Z skip: Need at least 2 CUDA devices (1.311s)
2022-12-01T10:13:14.6235897Z   test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_sharded_state_dict_checkpoint_wrap_first (__main__.TestFSDPStateDict)
2022-12-01T10:13:14.6236460Z Tests saving the state dict, zeroing a target model's parameters, and ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7044
2022-12-01T10:13:14.6236859Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7045
2022-12-01T10:13:14.6237315Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:13:14.6237633Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:13:14.6238064Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:13:14.6238414Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:13:14.6238741Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:13:14.6239210Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:13:14.6239539Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:13:14.6239966Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:13:14.6240298Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:13:14.6240639Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:13:14.6241138Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:13:14.6241656Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:13:14.6242027Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:13:14.6242372Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:13:14.6242630Z dist init r=0, world=2
2022-12-01T10:13:14.6242799Z dist init r=1, world=2
2022-12-01T10:13:14.6243009Z skip: Need at least 2 CUDA devices (1.210s)
2022-12-01T10:13:14.6243358Z   test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_sharded_state_dict_checkpoint_wrap_second (__main__.TestFSDPStateDict)
2022-12-01T10:13:14.6243929Z Tests saving the state dict, zeroing a target model's parameters, and ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7072
2022-12-01T10:13:14.6244317Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7073
2022-12-01T10:13:14.6244768Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:13:14.6245096Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:13:14.6245511Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:13:14.6245859Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:13:14.6246194Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:13:14.6246661Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:13:14.6247009Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:13:14.6247486Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:13:14.6247831Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:13:14.6248168Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:13:14.6248644Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:13:14.6249165Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:13:14.6249551Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:13:14.6249887Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:13:14.6250154Z dist init r=0, world=2
2022-12-01T10:13:14.6250338Z dist init r=1, world=2
2022-12-01T10:13:14.6250545Z skip: Need at least 2 CUDA devices (1.310s)
2022-12-01T10:13:14.6250868Z   test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_state_dict_checkpoint_wrap_both (__main__.TestFSDPStateDict)
2022-12-01T10:13:14.6251424Z Tests saving the state dict, zeroing a target model's parameters, and ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7100
2022-12-01T10:13:14.6251820Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7101
2022-12-01T10:13:14.6252260Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:13:14.6252586Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:13:14.6253015Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:13:14.6253363Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:13:14.6253688Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:13:14.6254155Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:13:14.6254483Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:13:14.6254899Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:13:14.6255241Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:13:14.6255576Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:13:14.6256065Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:13:14.6256575Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:13:14.6256968Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:13:14.6257317Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:13:14.6257580Z dist init r=0, world=2
2022-12-01T10:13:14.6257747Z dist init r=1, world=2
2022-12-01T10:13:14.6257956Z skip: Need at least 2 CUDA devices (1.311s)
2022-12-01T10:13:14.6258377Z   test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_state_dict_checkpoint_wrap_first (__main__.TestFSDPStateDict)
2022-12-01T10:13:14.6258973Z Tests saving the state dict, zeroing a target model's parameters, and ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7128
2022-12-01T10:13:14.6259367Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7129
2022-12-01T10:13:14.6259866Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:13:14.6260226Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:13:14.6260787Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:13:14.6261133Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:13:14.6261470Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:13:14.6261922Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:13:14.6262248Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:13:14.6262676Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:13:14.6263022Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:13:14.6263342Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:13:14.6263841Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:13:14.6264362Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:13:14.6264749Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:13:14.6265082Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:13:14.6265337Z dist init r=1, world=2
2022-12-01T10:13:14.6265524Z dist init r=0, world=2
2022-12-01T10:13:14.6265718Z skip: Need at least 2 CUDA devices (1.311s)
2022-12-01T10:13:14.6266059Z   test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_state_dict_checkpoint_wrap_second (__main__.TestFSDPStateDict)
2022-12-01T10:13:14.6266617Z Tests saving the state dict, zeroing a target model's parameters, and ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7156
2022-12-01T10:13:14.6267018Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7157
2022-12-01T10:13:14.6267450Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:13:14.6267778Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:13:14.6268206Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:13:14.6268539Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:13:14.6268876Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:13:14.6269347Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:13:14.6269677Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:13:14.6270091Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:13:14.6270432Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:13:14.6270770Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:13:14.6271262Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:13:14.6271768Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:13:14.6272150Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:13:14.6272560Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:13:14.6272810Z dist init r=0, world=2
2022-12-01T10:13:14.6273029Z dist init r=1, world=2
2022-12-01T10:13:14.6273239Z skip: Need at least 2 CUDA devices (1.311s)
2022-12-01T10:13:14.6273610Z   test_save_and_load_after_forward_state_dict_state_dict_type_local_state_dict_mixed_precision_False_state_dict_rank0_and_offload_False (__main__.TestFSDPStateDict)
2022-12-01T10:13:14.6274071Z Test that saving after some training results in params being updated as ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7184
2022-12-01T10:13:14.6274467Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7185
2022-12-01T10:13:14.6274920Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:13:14.6275236Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:13:14.6275661Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:13:14.6276009Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:13:14.6276344Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:13:14.6276804Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:13:14.6277132Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:13:14.6277557Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:13:14.6277902Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:13:14.6278224Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:13:14.6278718Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:13:14.6279240Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:13:14.6279616Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:13:14.6279966Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:13:14.6280220Z dist init r=0, world=2
2022-12-01T10:13:14.6280405Z dist init r=1, world=2
2022-12-01T10:13:14.6280601Z skip: Need at least 2 CUDA devices (1.312s)
2022-12-01T10:13:14.6280970Z   test_save_and_load_after_forward_state_dict_state_dict_type_local_state_dict_mixed_precision_False_state_dict_rank0_and_offload_True (__main__.TestFSDPStateDict)
2022-12-01T10:13:14.6281449Z Test that saving after some training results in params being updated as ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7212
2022-12-01T10:13:14.6281833Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7213
2022-12-01T10:13:14.6282287Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:13:14.6282615Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:13:14.6283044Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:13:14.6283375Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:13:14.6283713Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:13:14.6284181Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:13:14.6284510Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:13:14.6284956Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:13:14.6285330Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:13:14.6285666Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:13:14.6286146Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:13:14.6286662Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:13:14.6287045Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:13:14.6287390Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:13:14.6287635Z dist init r=1, world=2
2022-12-01T10:13:14.6287824Z dist init r=0, world=2
2022-12-01T10:13:14.6288032Z skip: Need at least 2 CUDA devices (1.312s)
2022-12-01T10:13:14.6288389Z   test_save_and_load_after_forward_state_dict_state_dict_type_local_state_dict_mixed_precision_True_state_dict_rank0_and_offload_False (__main__.TestFSDPStateDict)
2022-12-01T10:13:14.6288871Z Test that saving after some training results in params being updated as ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7240
2022-12-01T10:13:14.6289268Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7241
2022-12-01T10:13:14.6289717Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:13:14.6290030Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:13:14.6290456Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:13:14.6290801Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:13:14.6291139Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:13:14.6291593Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:13:14.6291924Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:13:14.6292351Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:13:14.6292679Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:13:14.6293011Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:13:14.6293506Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:13:14.6294023Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:13:14.6294400Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:13:14.6294751Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:13:14.6295009Z dist init r=0, world=2
2022-12-01T10:13:14.6295179Z dist init r=1, world=2
2022-12-01T10:13:14.6295387Z skip: Need at least 2 CUDA devices (1.312s)
2022-12-01T10:13:14.6295750Z   test_save_and_load_after_forward_state_dict_state_dict_type_local_state_dict_mixed_precision_True_state_dict_rank0_and_offload_True (__main__.TestFSDPStateDict)
2022-12-01T10:13:14.6296225Z Test that saving after some training results in params being updated as ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7268
2022-12-01T10:13:14.6296610Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7269
2022-12-01T10:13:14.6297059Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:13:14.6297418Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:13:14.6297874Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:13:14.6298295Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:13:14.6298638Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:13:14.6299119Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:13:14.6299433Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:13:14.6299861Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:13:14.6300211Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:13:14.6300757Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:13:14.6301246Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:13:14.6301768Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:13:14.6302152Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:13:14.6302498Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:13:14.6302743Z dist init r=0, world=2
2022-12-01T10:13:14.6302929Z dist init r=1, world=2
2022-12-01T10:13:14.6303137Z skip: Need at least 2 CUDA devices (1.312s)
2022-12-01T10:13:14.6303494Z   test_save_and_load_after_forward_state_dict_state_dict_type_sharded_state_dict_mixed_precision_False_state_dict_rank0_and_offload_False (__main__.TestFSDPStateDict)
2022-12-01T10:13:14.6303980Z Test that saving after some training results in params being updated as ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7296
2022-12-01T10:13:14.6304383Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7297
2022-12-01T10:13:14.6304835Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:13:14.6305153Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:13:14.6305580Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:13:14.6305925Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:13:14.6306248Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:13:14.6306719Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:13:14.6307049Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:13:14.6307475Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:13:14.6307804Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:13:14.6308140Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:13:14.6308635Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:13:14.6309155Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:13:14.6309525Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:13:14.6309873Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:13:14.6310131Z dist init r=1, world=2
2022-12-01T10:13:14.6310366Z dist init r=0, world=2
2022-12-01T10:13:14.6310614Z skip: Need at least 2 CUDA devices (1.312s)
2022-12-01T10:13:14.6310983Z   test_save_and_load_after_forward_state_dict_state_dict_type_sharded_state_dict_mixed_precision_False_state_dict_rank0_and_offload_True (__main__.TestFSDPStateDict)
2022-12-01T10:13:14.6311459Z Test that saving after some training results in params being updated as ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7324
2022-12-01T10:13:14.6311840Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7325
2022-12-01T10:13:14.6312292Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:13:14.6312619Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:13:14.6313036Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:13:14.6313387Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:13:14.6313727Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:13:14.6314197Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:13:14.6314507Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:13:14.6314935Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:13:14.6315280Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:13:14.6315619Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:13:14.6316100Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:13:14.6316621Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:13:14.6317014Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:13:14.6317350Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:13:14.6317609Z dist init r=0, world=2
2022-12-01T10:13:14.6317792Z dist init r=1, world=2
2022-12-01T10:13:14.6318001Z skip: Need at least 2 CUDA devices (1.312s)
2022-12-01T10:13:14.6318360Z   test_save_and_load_after_forward_state_dict_state_dict_type_sharded_state_dict_mixed_precision_True_state_dict_rank0_and_offload_False (__main__.TestFSDPStateDict)
2022-12-01T10:13:14.6318843Z Test that saving after some training results in params being updated as ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7352
2022-12-01T10:13:14.6319237Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7353
2022-12-01T10:13:14.6319674Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:13:14.6320007Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:13:14.6320432Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:13:14.6320782Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:13:14.6321107Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:13:14.6321572Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:13:14.6321899Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:13:14.6322327Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:13:14.6322693Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:13:14.6323056Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:13:14.6323547Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:13:14.6324052Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:13:14.6324439Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:13:14.6324786Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:13:14.6325044Z dist init r=1, world=2
2022-12-01T10:13:14.6325210Z dist init r=0, world=2
2022-12-01T10:13:14.6325415Z skip: Need at least 2 CUDA devices (1.312s)
2022-12-01T10:13:14.6325788Z   test_save_and_load_after_forward_state_dict_state_dict_type_sharded_state_dict_mixed_precision_True_state_dict_rank0_and_offload_True (__main__.TestFSDPStateDict)
2022-12-01T10:13:14.6326259Z Test that saving after some training results in params being updated as ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7380
2022-12-01T10:13:14.6326658Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7381
2022-12-01T10:13:14.6327110Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:13:14.6327439Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:13:14.6327854Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:13:14.6328200Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:13:14.6328536Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:13:14.6329003Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:13:14.6329319Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:13:14.6329745Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:13:14.6330087Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:13:14.6330407Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:13:14.6330900Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:13:14.6331414Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:13:14.6331797Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:13:14.6332130Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:13:14.6332384Z dist init r=0, world=2
2022-12-01T10:13:14.6332566Z dist init r=1, world=2
2022-12-01T10:13:14.6332761Z skip: Need at least 2 CUDA devices (1.312s)
2022-12-01T10:13:14.6333122Z   test_save_and_load_after_forward_state_dict_state_dict_type_state_dict_mixed_precision_False_state_dict_rank0_and_offload_False (__main__.TestFSDPStateDict)
2022-12-01T10:13:14.6333594Z Test that saving after some training results in params being updated as ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7408
2022-12-01T10:13:14.6333991Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7409
2022-12-01T10:13:14.6334427Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:13:14.6334757Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:13:14.6335224Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:13:14.6335610Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:13:14.6335934Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:13:14.6336400Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:13:14.6336730Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:13:14.6337143Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:13:14.6337486Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:13:14.6337820Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:13:14.6338403Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:13:14.6338921Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:13:14.6339304Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:13:14.6339650Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:13:14.6339909Z dist init r=0, world=2
2022-12-01T10:13:14.6340080Z dist init r=1, world=2
2022-12-01T10:13:14.6340289Z skip: Need at least 2 CUDA devices (1.312s)
2022-12-01T10:13:14.6340822Z   test_save_and_load_after_forward_state_dict_state_dict_type_state_dict_mixed_precision_False_state_dict_rank0_and_offload_True (__main__.TestFSDPStateDict)
2022-12-01T10:13:14.6341283Z Test that saving after some training results in params being updated as ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7436
2022-12-01T10:13:14.6341680Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7437
2022-12-01T10:13:14.6342138Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:13:14.6342466Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:13:14.6342884Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:13:14.6343228Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:13:14.6343565Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:13:14.6344019Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:13:14.6344348Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:13:14.6344777Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:13:14.6345124Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:13:14.6345444Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:13:14.6345942Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:13:14.6346459Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:13:14.6346844Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:13:14.6347175Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:13:14.6347430Z dist init r=1, world=2
2022-12-01T10:13:14.6347613Z dist init r=0, world=2
2022-12-01T10:13:14.6347874Z skip: Need at least 2 CUDA devices (1.312s)
2022-12-01T10:13:14.6348235Z   test_save_and_load_after_forward_state_dict_state_dict_type_state_dict_mixed_precision_True_state_dict_rank0_and_offload_False (__main__.TestFSDPStateDict)
2022-12-01T10:13:14.6348741Z Test that saving after some training results in params being updated as ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7464
2022-12-01T10:13:14.6349136Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7465
2022-12-01T10:13:14.6349574Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:13:14.6349905Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:13:14.6350332Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:13:14.6350664Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:13:14.6350998Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:13:14.6351470Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:13:14.6351793Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:13:14.6352202Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:13:14.6352547Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:13:14.6352883Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:13:14.6353375Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:13:14.6353875Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:13:14.6354266Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:13:14.6354616Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:13:14.6354862Z dist init r=0, world=2
2022-12-01T10:13:14.6355042Z dist init r=1, world=2
2022-12-01T10:13:14.6355252Z skip: Need at least 2 CUDA devices (1.312s)
2022-12-01T10:13:14.6355614Z   test_save_and_load_after_forward_state_dict_state_dict_type_state_dict_mixed_precision_True_state_dict_rank0_and_offload_True (__main__.TestFSDPStateDict)
2022-12-01T10:13:14.6356075Z Test that saving after some training results in params being updated as ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7492
2022-12-01T10:13:14.6356469Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7493
2022-12-01T10:13:14.6356920Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:13:14.6357241Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:13:14.6357671Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:13:14.6358017Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:13:14.6358352Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:13:14.6358806Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:13:14.6359131Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:13:14.6359555Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:13:14.6359901Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:13:14.6360260Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:13:14.6360782Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:13:14.6361304Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:13:14.6361678Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:13:14.6362024Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:13:14.6362276Z dist init r=1, world=2
2022-12-01T10:13:14.6362458Z dist init r=0, world=2
2022-12-01T10:13:14.6362655Z skip: Need at least 2 CUDA devices (1.312s)
2022-12-01T10:13:14.6363014Z   test_state_dict_load_into_local_module_state_dict_type_sharded_state_dict_state_dict_rank0_and_offload_False_fsdp_root_False (__main__.TestFSDPStateDict)
2022-12-01T10:13:14.6363583Z Tests that FSDP's state_dict can be loaded into a local model. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7520
2022-12-01T10:13:14.6363961Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7521
2022-12-01T10:13:14.6364413Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:13:14.6364743Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:13:14.6365172Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:13:14.6365505Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:13:14.6365840Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:13:14.6366306Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:13:14.6366636Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:13:14.6367050Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:13:14.6367396Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:13:14.6367729Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:13:14.6368207Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:13:14.6368730Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:13:14.6369114Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:13:14.6369461Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:13:14.6369707Z dist init r=0, world=2
2022-12-01T10:13:14.6369891Z dist init r=1, world=2
2022-12-01T10:13:14.6370123Z skip: Need at least 2 CUDA devices (1.312s)
2022-12-01T10:13:14.6370469Z   test_state_dict_load_into_local_module_state_dict_type_sharded_state_dict_state_dict_rank0_and_offload_False_fsdp_root_True (__main__.TestFSDPStateDict)
2022-12-01T10:13:14.6371029Z Tests that FSDP's state_dict can be loaded into a local model. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7548
2022-12-01T10:13:14.6371416Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7549
2022-12-01T10:13:14.6371866Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:13:14.6372239Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:13:14.6372666Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:13:14.6373042Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:13:14.6373407Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:13:14.6373861Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:13:14.6374190Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:13:14.6374616Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:13:14.6374945Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:13:14.6375281Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:13:14.6375778Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:13:14.6376304Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:13:14.6376676Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:13:14.6377027Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:13:14.6377285Z dist init r=0, world=2
2022-12-01T10:13:14.6377451Z dist init r=1, world=2
2022-12-01T10:13:14.6377659Z skip: Need at least 2 CUDA devices (1.212s)
2022-12-01T10:13:14.6378020Z   test_state_dict_load_into_local_module_state_dict_type_sharded_state_dict_state_dict_rank0_and_offload_True_fsdp_root_False (__main__.TestFSDPStateDict)
2022-12-01T10:13:14.6378672Z Tests that FSDP's state_dict can be loaded into a local model. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7576
2022-12-01T10:13:14.6379051Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7577
2022-12-01T10:13:14.6379502Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:13:14.6379830Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:13:14.6380258Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:13:14.6380712Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:13:14.6381059Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:13:14.6381529Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:13:14.6381840Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:13:14.6382266Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:13:14.6382616Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:13:14.6382953Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:13:14.6383432Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:13:14.6383951Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:13:14.6384336Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:13:14.6384684Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:13:14.6384928Z dist init r=0, world=2
2022-12-01T10:13:14.6385109Z dist init r=1, world=2
2022-12-01T10:13:14.6385316Z skip: Need at least 2 CUDA devices (1.312s)
2022-12-01T10:13:14.6385721Z   test_state_dict_load_into_local_module_state_dict_type_sharded_state_dict_state_dict_rank0_and_offload_True_fsdp_root_True (__main__.TestFSDPStateDict)
2022-12-01T10:13:14.6386313Z Tests that FSDP's state_dict can be loaded into a local model. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7604
2022-12-01T10:13:14.6386701Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7605
2022-12-01T10:13:14.6387247Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:13:14.6387591Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:13:14.6388020Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:13:14.6388365Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:13:14.6388685Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:13:14.6389153Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:13:14.6389482Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:13:14.6389909Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:13:14.6390235Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:13:14.6390570Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:13:14.6391063Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:13:14.6391581Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:13:14.6391948Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:13:14.6392300Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:13:14.6392560Z dist init r=1, world=2
2022-12-01T10:13:14.6392769Z dist init r=0, world=2
2022-12-01T10:13:14.6392978Z skip: Need at least 2 CUDA devices (1.312s)
2022-12-01T10:13:14.6393333Z   test_state_dict_load_into_local_module_state_dict_type_state_dict_state_dict_rank0_and_offload_False_fsdp_root_False (__main__.TestFSDPStateDict)
2022-12-01T10:13:14.6393874Z Tests that FSDP's state_dict can be loaded into a local model. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7632
2022-12-01T10:13:14.6394266Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7633
2022-12-01T10:13:14.6394717Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:13:14.6395048Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:13:14.6395467Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:13:14.6395813Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:13:14.6396150Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:13:14.6396619Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:13:14.6396931Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:13:14.6397358Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:13:14.6397707Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:13:14.6398147Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:13:14.6398798Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:13:14.6399356Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:13:14.6399742Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:13:14.6400082Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:13:14.6400335Z dist init r=1, world=2
2022-12-01T10:13:14.6400520Z dist init r=0, world=2
2022-12-01T10:13:14.6400716Z skip: Need at least 2 CUDA devices (1.312s)
2022-12-01T10:13:14.6401067Z   test_state_dict_load_into_local_module_state_dict_type_state_dict_state_dict_rank0_and_offload_False_fsdp_root_True (__main__.TestFSDPStateDict)
2022-12-01T10:13:14.6401622Z Tests that FSDP's state_dict can be loaded into a local model. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7660
2022-12-01T10:13:14.6402012Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7661
2022-12-01T10:13:14.6402453Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:13:14.6402785Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:13:14.6403216Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:13:14.6403563Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:13:14.6403887Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:13:14.6404355Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:13:14.6404681Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:13:14.6405097Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:13:14.6405444Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:13:14.6405782Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:13:14.6406272Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:13:14.6406782Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:13:14.6407164Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:13:14.6407516Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:13:14.6407775Z dist init r=1, world=2
2022-12-01T10:13:14.6407940Z dist init r=0, world=2
2022-12-01T10:13:14.6408149Z skip: Need at least 2 CUDA devices (1.312s)
2022-12-01T10:13:14.6408507Z   test_state_dict_load_into_local_module_state_dict_type_state_dict_state_dict_rank0_and_offload_True_fsdp_root_False (__main__.TestFSDPStateDict)
2022-12-01T10:13:14.6409049Z Tests that FSDP's state_dict can be loaded into a local model. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7688
2022-12-01T10:13:14.6409437Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7689
2022-12-01T10:13:14.6409712Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:13:14.6409836Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:13:14.6410117Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:13:14.6410254Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:13:14.6410467Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:13:14.6410727Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:13:14.6410912Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:13:14.6411192Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:13:14.6411326Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:13:14.6411503Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:13:14.6411800Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:13:14.6412095Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:13:14.6412262Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:13:14.6412425Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:13:14.6412492Z dist init r=1, world=2
2022-12-01T10:13:14.6412567Z dist init r=0, world=2
2022-12-01T10:13:14.6412737Z skip: Need at least 2 CUDA devices (1.312s)
2022-12-01T10:13:14.6412952Z   test_state_dict_load_into_local_module_state_dict_type_state_dict_state_dict_rank0_and_offload_True_fsdp_root_True (__main__.TestFSDPStateDict)
2022-12-01T10:13:14.6413273Z Tests that FSDP's state_dict can be loaded into a local model. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7716
2022-12-01T10:13:14.6413426Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7717
2022-12-01T10:13:14.6413701Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:13:14.6413825Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:13:14.6414094Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:13:14.6414233Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:13:14.6414409Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:13:14.6414679Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:13:14.6414803Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:13:14.6415080Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:13:14.6415215Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:13:14.6415393Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:13:14.6415679Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:13:14.6415975Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:13:14.6416142Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:13:14.6416306Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:13:14.6416382Z dist init r=1, world=2
2022-12-01T10:13:14.6416454Z dist init r=0, world=2
2022-12-01T10:13:14.6416557Z skip: Need at least 2 CUDA devices (1.312s)
2022-12-01T10:13:14.6416704Z   test_state_dict_rank0_offload_save_load_flow (__main__.TestFSDPStateDict)
2022-12-01T10:13:14.6416915Z Tests saving a model checkpoint only on rank 0 and loading it only ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7744
2022-12-01T10:13:14.6417101Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7745
2022-12-01T10:13:14.6417415Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:13:14.6417537Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:13:14.6417821Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:13:14.6417957Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:13:14.6418223Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:13:14.6418499Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:13:14.6418622Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:13:14.6418888Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:13:14.6419024Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:13:14.6419199Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:13:14.6419499Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:13:14.6419793Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:13:14.6419961Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:13:14.6420124Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:13:14.6420199Z dist init r=1, world=2
2022-12-01T10:13:14.6420273Z dist init r=0, world=2
2022-12-01T10:13:14.6420477Z skip: Need at least 2 CUDA devices (1.311s)
2022-12-01T10:13:14.6420736Z   test_state_dict_save_load_flow_state_dict_type_local_state_dict (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7772
2022-12-01T10:13:14.6420896Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7773
2022-12-01T10:13:14.6421169Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:13:14.6421293Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:13:14.6421572Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:13:14.6421707Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:13:14.6421883Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:13:14.6422141Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:13:14.6422267Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:13:14.6422549Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:13:14.6422684Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:13:14.6422860Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:13:14.6423158Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:13:14.6423453Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:13:14.6423616Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:13:14.6423841Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:13:14.6423905Z dist init r=1, world=2
2022-12-01T10:13:14.6424015Z dist init r=0, world=2
2022-12-01T10:13:14.6424119Z skip: Need at least 2 CUDA devices (1.310s)
2022-12-01T10:13:14.6424375Z   test_state_dict_save_load_flow_state_dict_type_sharded_state_dict (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7800
2022-12-01T10:13:14.6424527Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7801
2022-12-01T10:13:14.6424803Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:13:14.6424928Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:13:14.6425210Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:13:14.6425332Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:13:14.6425509Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:13:14.6425781Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:13:14.6425903Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:13:14.6426182Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:13:14.6426318Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:13:14.6426497Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:13:14.6426798Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:13:14.6427093Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:13:14.6427247Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:13:14.6427412Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:13:14.6427487Z dist init r=1, world=2
2022-12-01T10:13:14.6427560Z dist init r=0, world=2
2022-12-01T10:13:14.6427663Z skip: Need at least 2 CUDA devices (1.310s)
2022-12-01T10:13:14.6427911Z   test_state_dict_save_load_flow_state_dict_type_state_dict (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7828
2022-12-01T10:13:14.6428067Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7829
2022-12-01T10:13:14.6428340Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:13:14.6428450Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:13:14.6428735Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:13:14.6428934Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:13:14.6429137Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:13:14.6429412Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:13:14.6429536Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:13:14.6429817Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:13:14.6429950Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:13:14.6430124Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:13:14.6430442Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:13:14.6430818Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:13:14.6430982Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:13:14.6431146Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:13:14.6431221Z dist init r=1, world=2
2022-12-01T10:13:14.6431294Z dist init r=0, world=2
2022-12-01T10:13:14.6431398Z skip: Need at least 2 CUDA devices (1.310s)
2022-12-01T10:13:14.6431667Z   test_state_dict_skip_module_state_dict_type_local_state_dict_double_nest_True (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7856
2022-12-01T10:13:14.6431808Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7857
2022-12-01T10:13:14.6432085Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:13:14.6432211Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:13:14.6432490Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:13:14.6432625Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:13:14.6432805Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:13:14.6433076Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:13:14.6433198Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:13:14.6433477Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:13:14.6433601Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:13:14.6433780Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:13:14.6434080Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:13:14.6434377Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:13:14.6434540Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:13:14.6434706Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:13:14.6434781Z dist init r=0, world=2
2022-12-01T10:13:14.6434853Z dist init r=1, world=2
2022-12-01T10:13:14.6434943Z skip: Need at least 2 CUDA devices (1.312s)
2022-12-01T10:13:14.6435220Z   test_state_dict_skip_module_state_dict_type_sharded_state_dict_double_nest_True (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7884
2022-12-01T10:13:14.6435377Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7885
2022-12-01T10:13:14.6435649Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:13:14.6435772Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:13:14.6436051Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:13:14.6436186Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:13:14.6436360Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:13:14.6436628Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:13:14.6436737Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:13:14.6437046Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:13:14.6437207Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:13:14.6437382Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:13:14.6437681Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:13:14.6437977Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:13:14.6438143Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:13:14.6438307Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:13:14.6438369Z dist init r=0, world=2
2022-12-01T10:13:14.6438442Z dist init r=1, world=2
2022-12-01T10:13:14.6438548Z skip: Need at least 2 CUDA devices (1.312s)
2022-12-01T10:13:14.6438814Z   test_state_dict_skip_module_state_dict_type_state_dict_double_nest_True (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7912
2022-12-01T10:13:14.6438967Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7913
2022-12-01T10:13:14.6439238Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:13:14.6439360Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:13:14.6439763Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:13:14.6439901Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:13:14.6440129Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:13:14.6440479Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:13:14.6440605Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:13:14.6440886Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:13:14.6441020Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:13:14.6441198Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:13:14.6441499Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:13:14.6441795Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:13:14.6441948Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:13:14.6442112Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:13:14.6442189Z dist init r=1, world=2
2022-12-01T10:13:14.6442266Z dist init r=0, world=2
2022-12-01T10:13:14.6442370Z skip: Need at least 2 CUDA devices (1.312s)
2022-12-01T10:13:14.6442586Z   test_state_dict_type (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7940
2022-12-01T10:13:14.6442739Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7941
2022-12-01T10:13:14.6443012Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:13:14.6443122Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:13:14.6443402Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:13:14.6443570Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:13:14.6443775Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:13:14.6444041Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:13:14.6444162Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:13:14.6444440Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:13:14.6444577Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:13:14.6444750Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:13:14.6445034Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:13:14.6445330Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:13:14.6445494Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:13:14.6445659Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:13:14.6445735Z dist init r=0, world=2
2022-12-01T10:13:14.6445808Z dist init r=1, world=2
2022-12-01T10:13:14.6445911Z skip: Need at least 2 CUDA devices (1.311s)
2022-12-01T10:13:14.6446207Z   test_state_dict_with_ignored_modules_state_dict_type_sharded_state_dict_prefix_False_ignore_inner_False (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7968
2022-12-01T10:13:14.6446348Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7969
2022-12-01T10:13:14.6446618Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:13:14.6446743Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:13:14.6447025Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:13:14.6447160Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:13:14.6447334Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:13:14.6447609Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:13:14.6447734Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:13:14.6448015Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:13:14.6448137Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:13:14.6448313Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:13:14.6448608Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:13:14.6448907Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:13:14.6449072Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:13:14.6449235Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:13:14.6449309Z dist init r=0, world=2
2022-12-01T10:13:14.6449383Z dist init r=1, world=2
2022-12-01T10:13:14.6449473Z skip: Need at least 2 CUDA devices (1.312s)
2022-12-01T10:13:14.6449767Z   test_state_dict_with_ignored_modules_state_dict_type_sharded_state_dict_prefix_False_ignore_inner_True (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7996
2022-12-01T10:13:14.6449954Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7997
2022-12-01T10:13:14.6450254Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:13:14.6450377Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:13:14.6450659Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:13:14.6450828Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:13:14.6451082Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:13:14.6451359Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:13:14.6451469Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:13:14.6451749Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:13:14.6451886Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:13:14.6452061Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:13:14.6452357Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:13:14.6452654Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:13:14.6452819Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:13:14.6452981Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:13:14.6453055Z dist init r=1, world=2
2022-12-01T10:13:14.6453115Z dist init r=0, world=2
2022-12-01T10:13:14.6453216Z skip: Need at least 2 CUDA devices (1.312s)
2022-12-01T10:13:14.6453510Z   test_state_dict_with_ignored_modules_state_dict_type_sharded_state_dict_prefix_True_ignore_inner_False (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8024
2022-12-01T10:13:14.6453667Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8025
2022-12-01T10:13:14.6453938Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:13:14.6454063Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:13:14.6454343Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:13:14.6454479Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:13:14.6454641Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:13:14.6454912Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:13:14.6455036Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:13:14.6455312Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:13:14.6455446Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:13:14.6455622Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:13:14.6455919Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:13:14.6456215Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:13:14.6456378Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:13:14.6456562Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:13:14.6456665Z dist init r=1, world=2
2022-12-01T10:13:14.6456737Z dist init r=0, world=2
2022-12-01T10:13:14.6456841Z skip: Need at least 2 CUDA devices (1.312s)
2022-12-01T10:13:14.6457135Z   test_state_dict_with_ignored_modules_state_dict_type_sharded_state_dict_prefix_True_ignore_inner_True (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8052
2022-12-01T10:13:14.6457288Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8053
2022-12-01T10:13:14.6457563Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:13:14.6457689Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:13:14.6457954Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:13:14.6458093Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:13:14.6458342Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:13:14.6458621Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:13:14.6458746Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:13:14.6459025Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:13:14.6459161Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:13:14.6459336Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:13:14.6459637Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:13:14.6459923Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:13:14.6460090Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:13:14.6460251Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:13:14.6460326Z dist init r=1, world=2
2022-12-01T10:13:14.6460481Z dist init r=0, world=2
2022-12-01T10:13:14.6460587Z skip: Need at least 2 CUDA devices (1.312s)
2022-12-01T10:13:14.6460877Z   test_state_dict_with_ignored_modules_state_dict_type_state_dict_prefix_False_ignore_inner_False (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8080
2022-12-01T10:13:14.6461031Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8081
2022-12-01T10:13:14.6461292Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:13:14.6461417Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:13:14.6461841Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:13:14.6461991Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:13:14.6462168Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:13:14.6462444Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:13:14.6462568Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:13:14.6462846Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:13:14.6462982Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:13:14.6463209Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:13:14.6463513Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:13:14.6463866Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:13:14.6464030Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:13:14.6464193Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:13:14.6464269Z dist init r=0, world=2
2022-12-01T10:13:14.6464342Z dist init r=1, world=2
2022-12-01T10:13:14.6464448Z skip: Need at least 2 CUDA devices (1.312s)
2022-12-01T10:13:14.6464720Z   test_state_dict_with_ignored_modules_state_dict_type_state_dict_prefix_False_ignore_inner_True (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8108
2022-12-01T10:13:14.6464878Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8109
2022-12-01T10:13:14.6465154Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:13:14.6465282Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:13:14.6465563Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:13:14.6465700Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:13:14.6465876Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:13:14.6466149Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:13:14.6466273Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:13:14.6466539Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:13:14.6466679Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:13:14.6466859Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:13:14.6467158Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:13:14.6467452Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:13:14.6467618Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:13:14.6467777Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:13:14.6467853Z dist init r=1, world=2
2022-12-01T10:13:14.6467912Z dist init r=0, world=2
2022-12-01T10:13:14.6468015Z skip: Need at least 2 CUDA devices (1.312s)
2022-12-01T10:13:14.6468298Z   test_state_dict_with_ignored_modules_state_dict_type_state_dict_prefix_True_ignore_inner_False (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8136
2022-12-01T10:13:14.6468453Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8137
2022-12-01T10:13:14.6468730Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:13:14.6468853Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:13:14.6469133Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:13:14.6469267Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:13:14.6469445Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:13:14.6469730Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:13:14.6469882Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:13:14.6470162Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:13:14.6470297Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:13:14.6470472Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:13:14.6470770Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:13:14.6471064Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:13:14.6471229Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:13:14.6471395Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:13:14.6471459Z dist init r=1, world=2
2022-12-01T10:13:14.6471532Z dist init r=0, world=2
2022-12-01T10:13:14.6471636Z skip: Need at least 2 CUDA devices (1.312s)
2022-12-01T10:13:14.6471921Z   test_state_dict_with_ignored_modules_state_dict_type_state_dict_prefix_True_ignore_inner_True (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8164
2022-12-01T10:13:14.6472076Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8165
2022-12-01T10:13:14.6472351Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:13:14.6472472Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:13:14.6472755Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:13:14.6472877Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:13:14.6473053Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:13:14.6473325Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:13:14.6473450Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:13:14.6473734Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:13:14.6473868Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:13:14.6474043Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:13:14.6474340Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:13:14.6474639Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:13:14.6474792Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:13:14.6474952Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:13:14.6475028Z dist init r=1, world=2
2022-12-01T10:13:14.6475103Z dist init r=0, world=2
2022-12-01T10:13:14.6475206Z skip: Need at least 2 CUDA devices (1.312s)
2022-12-01T10:13:14.6475427Z   test_wrong_state_dict_config (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8192
2022-12-01T10:13:14.6475580Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8193
2022-12-01T10:13:14.6475853Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:13:14.6475964Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:13:14.6476270Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:13:14.6476433Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:13:14.6476611Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:13:14.6476884Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:13:14.6477008Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:13:14.6477289Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:13:14.6477425Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:13:14.6477599Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:13:14.6477884Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:13:14.6478180Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:13:14.6478343Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:13:14.6478506Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:13:14.6478582Z dist init r=1, world=2
2022-12-01T10:13:14.6478655Z dist init r=0, world=2
2022-12-01T10:13:14.6478758Z skip: Need at least 2 CUDA devices (1.310s)
2022-12-01T10:13:14.6478768Z 
2022-12-01T10:13:14.6478967Z ----------------------------------------------------------------------
2022-12-01T10:13:14.6479031Z Ran 70 tests in 91.352s
2022-12-01T10:13:14.6479037Z 
2022-12-01T10:13:14.6479108Z OK (skipped=70)
2022-12-01T10:13:14.6479114Z 
2022-12-01T10:13:14.6479199Z Generating XML reports...
2022-12-01T10:13:14.6479544Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_state_dict/TEST-TestFSDPStateDict-20221201101142.xml
2022-12-01T10:13:14.6479551Z 
2022-12-01T10:13:14.6479937Z ##[endgroup]
2022-12-01T10:13:14.6480303Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_state_dict (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_state_dict_3vynux9v)
2022-12-01T10:13:14.6480309Z 
2022-12-01T10:13:14.6480516Z Running distributed/fsdp/test_fsdp_optim_state ... [2022-12-01 10:13:14.595303]
2022-12-01T10:13:14.6480882Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_optim_state.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-12-01 10:13:14.595782]
2022-12-01T10:14:27.0875671Z 
2022-12-01T10:14:27.0905660Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_optim_state
2022-12-01T10:14:27.0906795Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_optim_state (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_optim_state_8meyz2ag)
2022-12-01T10:14:27.0917377Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_optim_state
2022-12-01T10:14:27.0927722Z 
2022-12-01T10:14:27.0927966Z Running tests...
2022-12-01T10:14:27.0928554Z ----------------------------------------------------------------------
2022-12-01T10:14:27.0929066Z   test_flatten_sharded_optim_state_dict_nested (__main__.TestFSDPOptimState)
2022-12-01T10:14:27.0929896Z Tests :meth:`flatten_sharded_optim_state_dict` for an FSDP-root ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8231
2022-12-01T10:14:27.0930537Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8232
2022-12-01T10:14:27.0931071Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8233
2022-12-01T10:14:27.0931624Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8234
2022-12-01T10:14:27.0932612Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:14:27.0933242Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:14:27.0933941Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:14:27.0934496Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:14:27.0935067Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:14:27.0935848Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:14:27.0936378Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:14:27.0937055Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:14:27.0937577Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:14:27.0938202Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:14:27.0939019Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:14:27.0939546Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:14:27.0940213Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:14:27.0940914Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:14:27.0941451Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:14:27.0942190Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:14:27.0942683Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:14:27.0943463Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:14:27.0944114Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:14:27.0944503Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-12-01T10:14:27.0945002Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:14:27.0945527Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:14:27.0946049Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:14:27.0946549Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:14:27.0946936Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:14:27.0947294Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:14:27.0947641Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:14:27.0947974Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:14:27.0948232Z dist init r=1, world=4
2022-12-01T10:14:27.0948421Z dist init r=0, world=4
2022-12-01T10:14:27.0948587Z dist init r=3, world=4
2022-12-01T10:14:27.0948765Z dist init r=2, world=4
2022-12-01T10:14:27.0948976Z skip: Need at least 2 CUDA devices (1.757s)
2022-12-01T10:14:27.0949256Z   test_flatten_sharded_optim_state_dict_transformer (__main__.TestFSDPOptimState)
2022-12-01T10:14:27.0949767Z Tests :meth:`flatten_sharded_optim_state_dict` for an FSDP-root ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8287
2022-12-01T10:14:27.0950282Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8288
2022-12-01T10:14:27.0950668Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8289
2022-12-01T10:14:27.0950981Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8290
2022-12-01T10:14:27.0951432Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:14:27.0951769Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:14:27.0952206Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:14:27.0952542Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:14:27.0952881Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:14:27.0953353Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:14:27.0953673Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:14:27.0954102Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:14:27.0954449Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:14:27.0954787Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:14:27.0955240Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:14:27.0955570Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:14:27.0955995Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:14:27.0956324Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:14:27.0956661Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:14:27.0957129Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:14:27.0957453Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:14:27.0957862Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:14:27.0958208Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:14:27.0958543Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-12-01T10:14:27.0959038Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:14:27.0959543Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:14:27.0960061Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:14:27.0960580Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:14:27.0960964Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:14:27.0961302Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:14:27.0961650Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:14:27.0961998Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:14:27.0962245Z dist init r=3, world=4
2022-12-01T10:14:27.0962429Z dist init r=2, world=4
2022-12-01T10:14:27.0962609Z dist init r=1, world=4
2022-12-01T10:14:27.0962773Z dist init r=0, world=4
2022-12-01T10:14:27.0963020Z skip: Need at least 2 CUDA devices (1.418s)
2022-12-01T10:14:27.0963319Z   test_full_optim_state_dict_keys (__main__.TestFSDPOptimState)
2022-12-01T10:14:27.0963711Z Tests that the parameter keys returned by ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8343
2022-12-01T10:14:27.0964086Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8344
2022-12-01T10:14:27.0964420Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8345
2022-12-01T10:14:27.0964749Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8346
2022-12-01T10:14:27.0965197Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:14:27.0965533Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:14:27.0965965Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:14:27.0966322Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:14:27.0966651Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:14:27.0967122Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:14:27.0967453Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:14:27.0967867Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:14:27.0968212Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:14:27.0968551Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:14:27.0969020Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:14:27.0969335Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:14:27.0969766Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:14:27.0970112Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:14:27.0970435Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:14:27.0970900Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:14:27.0971227Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:14:27.0971656Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:14:27.0971988Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:14:27.0972328Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-12-01T10:14:27.0972828Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:14:27.0973350Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:14:27.0973853Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:14:27.0974374Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:14:27.0974764Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:14:27.0975113Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:14:27.0975449Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:14:27.0975829Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:14:27.0976899Z dist init r=1, world=4
2022-12-01T10:14:27.0977073Z dist init r=3, world=4
2022-12-01T10:14:27.0977259Z dist init r=0, world=4
2022-12-01T10:14:27.0977439Z dist init r=2, world=4
2022-12-01T10:14:27.0977639Z skip: Need at least 2 CUDA devices (1.417s)
2022-12-01T10:14:27.0978007Z   test_full_optim_state_dict_nested_invalid (__main__.TestFSDPOptimState)
2022-12-01T10:14:27.0978408Z Tests that :meth:`full_optim_state_dict` raises an error when ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8399
2022-12-01T10:14:27.0978797Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8400
2022-12-01T10:14:27.0979111Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8401
2022-12-01T10:14:27.0979432Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8402
2022-12-01T10:14:27.0979897Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:14:27.0980222Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:14:27.0980805Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:14:27.0981153Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:14:27.0981494Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:14:27.0981951Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:14:27.0982280Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:14:27.0982711Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:14:27.0983060Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:14:27.0983382Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:14:27.0983853Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:14:27.0984183Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:14:27.0984594Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:14:27.0984938Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:14:27.0985279Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:14:27.0985754Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:14:27.0986068Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:14:27.0986497Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:14:27.0986848Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:14:27.0987173Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-12-01T10:14:27.0987675Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:14:27.0988195Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:14:27.0988711Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:14:27.0989207Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:14:27.0989685Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:14:27.0990090Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:14:27.0990438Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:14:27.0990769Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:14:27.0991030Z dist init r=3, world=4
2022-12-01T10:14:27.0991216Z dist init r=1, world=4
2022-12-01T10:14:27.0991385Z dist init r=2, world=4
2022-12-01T10:14:27.0991568Z dist init r=0, world=4
2022-12-01T10:14:27.0991779Z skip: Need at least 2 CUDA devices (1.417s)
2022-12-01T10:14:27.0992027Z   test_optim_input_warning (__main__.TestFSDPOptimState)
2022-12-01T10:14:27.0992412Z Tests that passing the ``optim_input`` argument into optimizer state ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8455
2022-12-01T10:14:27.0992811Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8456
2022-12-01T10:14:27.0993141Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8457
2022-12-01T10:14:27.0993451Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8458
2022-12-01T10:14:27.0993900Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:14:27.0994231Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:14:27.0994647Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:14:27.0994995Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:14:27.0995331Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:14:27.0995801Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:14:27.0996116Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:14:27.0996548Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:14:27.0996895Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:14:27.0997230Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:14:27.0997680Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:14:27.0998005Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:14:27.0998432Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:14:27.0998762Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:14:27.0999099Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:14:27.0999573Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:14:27.0999898Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:14:27.1000309Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:14:27.1000650Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:14:27.1000985Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-12-01T10:14:27.1001482Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:14:27.1001992Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:14:27.1002547Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:14:27.1003120Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:14:27.1003504Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:14:27.1003839Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:14:27.1004180Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:14:27.1004526Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:14:27.1004774Z dist init r=3, world=4
2022-12-01T10:14:27.1004958Z dist init r=0, world=4
2022-12-01T10:14:27.1005142Z dist init r=1, world=4
2022-12-01T10:14:27.1005308Z dist init r=2, world=4
2022-12-01T10:14:27.1005517Z skip: Need at least 2 CUDA devices (1.417s)
2022-12-01T10:14:27.1005913Z   test_optim_state_dict_nested_state_dict_type_StateDictType_FULL_STATE_DICT_use_multiple_param_groups_False_rank0_only_False_use_diff_optim_inputs_False (__main__.TestFSDPOptimState)
2022-12-01T10:14:27.1006413Z Tests :meth:`full_optim_state_dict` and `sharded_optim_state_dict` ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8511
2022-12-01T10:14:27.1006793Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8512
2022-12-01T10:14:27.1007120Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8513
2022-12-01T10:14:27.1007442Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8514
2022-12-01T10:14:27.1007882Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:14:27.1008214Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:14:27.1008647Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:14:27.1008997Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:14:27.1009324Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:14:27.1009797Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:14:27.1010124Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:14:27.1010552Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:14:27.1010882Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:14:27.1011218Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:14:27.1011687Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:14:27.1012005Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:14:27.1012428Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:14:27.1012773Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:14:27.1013109Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-12-01T10:14:27.1013561Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:14:27.1013887Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:14:27.1014307Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:14:27.1014635Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:14:27.1015034Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:14:27.1015566Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:14:27.1016089Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:14:27.1016599Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:14:27.1017115Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:14:27.1017506Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:14:27.1017858Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:14:27.1018282Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:14:27.1018626Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:14:27.1018889Z dist init r=3, world=4
2022-12-01T10:14:27.1019060Z dist init r=0, world=4
2022-12-01T10:14:27.1019246Z dist init r=2, world=4
2022-12-01T10:14:27.1019427Z dist init r=1, world=4
2022-12-01T10:14:27.1019622Z skip: Need at least 2 CUDA devices (1.417s)
2022-12-01T10:14:27.1020028Z   test_optim_state_dict_nested_state_dict_type_StateDictType_FULL_STATE_DICT_use_multiple_param_groups_False_rank0_only_False_use_diff_optim_inputs_True (__main__.TestFSDPOptimState)
2022-12-01T10:14:27.1020692Z Tests :meth:`full_optim_state_dict` and `sharded_optim_state_dict` ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8567
2022-12-01T10:14:27.1021094Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8568
2022-12-01T10:14:27.1021411Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8569
2022-12-01T10:14:27.1021742Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8570
2022-12-01T10:14:27.1022214Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:14:27.1022549Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:14:27.1022966Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:14:27.1023319Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:14:27.1023656Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:14:27.1024112Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:14:27.1024439Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:14:27.1024871Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:14:27.1025219Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:14:27.1025541Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-12-01T10:14:27.1026034Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:14:27.1026531Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:14:27.1026863Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:14:27.1027278Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:14:27.1027620Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:14:27.1028034Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:14:27.1028534Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:14:27.1028862Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:14:27.1029288Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:14:27.1029634Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:14:27.1029954Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:14:27.1030450Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:14:27.1030971Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:14:27.1031480Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:14:27.1031858Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:14:27.1032208Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:14:27.1032552Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:14:27.1032885Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:14:27.1033146Z dist init r=0, world=4
2022-12-01T10:14:27.1033330Z dist init r=2, world=4
2022-12-01T10:14:27.1033511Z dist init r=1, world=4
2022-12-01T10:14:27.1033674Z dist init r=3, world=4
2022-12-01T10:14:27.1033884Z skip: Need at least 2 CUDA devices (1.417s)
2022-12-01T10:14:27.1034278Z   test_optim_state_dict_nested_state_dict_type_StateDictType_FULL_STATE_DICT_use_multiple_param_groups_False_rank0_only_True_use_diff_optim_inputs_False (__main__.TestFSDPOptimState)
2022-12-01T10:14:27.1034762Z Tests :meth:`full_optim_state_dict` and `sharded_optim_state_dict` ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8623
2022-12-01T10:14:27.1035152Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8624
2022-12-01T10:14:27.1035479Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8625
2022-12-01T10:14:27.1035813Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8626
2022-12-01T10:14:27.1036251Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:14:27.1036583Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:14:27.1037011Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:14:27.1037347Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:14:27.1037688Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:14:27.1038162Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:14:27.1038494Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:14:27.1038909Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:14:27.1039253Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:14:27.1039590Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:14:27.1040058Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:14:27.1040414Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:14:27.1040871Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:14:27.1041217Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:14:27.1041538Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:14:27.1042005Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:14:27.1042335Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:14:27.1042766Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:14:27.1043097Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:14:27.1043434Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-12-01T10:14:27.1043929Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:14:27.1044437Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:14:27.1044954Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:14:27.1045465Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:14:27.1045850Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:14:27.1046181Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:14:27.1046528Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:14:27.1046870Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:14:27.1047133Z dist init r=0, world=4
2022-12-01T10:14:27.1047302Z dist init r=3, world=4
2022-12-01T10:14:27.1047482Z dist init r=1, world=4
2022-12-01T10:14:27.1047659Z dist init r=2, world=4
2022-12-01T10:14:27.1047853Z skip: Need at least 2 CUDA devices (1.417s)
2022-12-01T10:14:27.1048244Z   test_optim_state_dict_nested_state_dict_type_StateDictType_FULL_STATE_DICT_use_multiple_param_groups_False_rank0_only_True_use_diff_optim_inputs_True (__main__.TestFSDPOptimState)
2022-12-01T10:14:27.1048731Z Tests :meth:`full_optim_state_dict` and `sharded_optim_state_dict` ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8679
2022-12-01T10:14:27.1049124Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8680
2022-12-01T10:14:27.1049436Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8681
2022-12-01T10:14:27.1049755Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8682
2022-12-01T10:14:27.1050206Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:14:27.1050524Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:14:27.1050952Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:14:27.1051296Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:14:27.1051634Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:14:27.1052092Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:14:27.1052418Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:14:27.1052884Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:14:27.1053247Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:14:27.1053586Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-12-01T10:14:27.1054057Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:14:27.1054385Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:14:27.1054795Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:14:27.1055139Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:14:27.1055472Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:14:27.1055972Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:14:27.1056461Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:14:27.1056794Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:14:27.1057225Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:14:27.1057558Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:14:27.1057971Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:14:27.1058479Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:14:27.1059002Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:14:27.1059505Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:14:27.1059898Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:14:27.1060253Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:14:27.1060746Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:14:27.1061079Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:14:27.1061336Z dist init r=3, world=4
2022-12-01T10:14:27.1061526Z dist init r=1, world=4
2022-12-01T10:14:27.1061695Z dist init r=0, world=4
2022-12-01T10:14:27.1061876Z dist init r=2, world=4
2022-12-01T10:14:27.1062086Z skip: Need at least 2 CUDA devices (1.417s)
2022-12-01T10:14:27.1062472Z   test_optim_state_dict_nested_state_dict_type_StateDictType_FULL_STATE_DICT_use_multiple_param_groups_True_rank0_only_False_use_diff_optim_inputs_False (__main__.TestFSDPOptimState)
2022-12-01T10:14:27.1062969Z Tests :meth:`full_optim_state_dict` and `sharded_optim_state_dict` ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8735
2022-12-01T10:14:27.1063362Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8736
2022-12-01T10:14:27.1063689Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8737
2022-12-01T10:14:27.1063996Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8738
2022-12-01T10:14:27.1064449Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:14:27.1064782Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:14:27.1065209Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:14:27.1065542Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:14:27.1065953Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:14:27.1066463Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:14:27.1066782Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:14:27.1067211Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:14:27.1067559Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:14:27.1067897Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:14:27.1068354Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:14:27.1068683Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:14:27.1069119Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:14:27.1069456Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:14:27.1069793Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-12-01T10:14:27.1070288Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:14:27.1070786Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:14:27.1071100Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:14:27.1071524Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:14:27.1071866Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:14:27.1072200Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:14:27.1072685Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:14:27.1073205Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:14:27.1073721Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:14:27.1074105Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:14:27.1074438Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:14:27.1074777Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:14:27.1075125Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:14:27.1075373Z dist init r=1, world=4
2022-12-01T10:14:27.1075558Z dist init r=0, world=4
2022-12-01T10:14:27.1075739Z dist init r=2, world=4
2022-12-01T10:14:27.1075904Z dist init r=3, world=4
2022-12-01T10:14:27.1076118Z skip: Need at least 2 CUDA devices (1.417s)
2022-12-01T10:14:27.1076509Z   test_optim_state_dict_nested_state_dict_type_StateDictType_FULL_STATE_DICT_use_multiple_param_groups_True_rank0_only_False_use_diff_optim_inputs_True (__main__.TestFSDPOptimState)
2022-12-01T10:14:27.1077002Z Tests :meth:`full_optim_state_dict` and `sharded_optim_state_dict` ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8791
2022-12-01T10:14:27.1077378Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8792
2022-12-01T10:14:27.1077746Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8793
2022-12-01T10:14:27.1078163Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8794
2022-12-01T10:14:27.1078651Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:14:27.1079028Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:14:27.1079460Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:14:27.1079812Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:14:27.1080142Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:14:27.1080614Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:14:27.1080945Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:14:27.1081371Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:14:27.1081709Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:14:27.1082045Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:14:27.1082512Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:14:27.1082826Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:14:27.1083252Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:14:27.1083593Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:14:27.1083926Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-12-01T10:14:27.1084404Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:14:27.1084903Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:14:27.1085230Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:14:27.1085658Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:14:27.1085989Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:14:27.1086326Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:14:27.1086837Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:14:27.1087336Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:14:27.1087850Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:14:27.1088238Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:14:27.1088592Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:14:27.1088926Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:14:27.1089262Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:14:27.1089523Z dist init r=2, world=4
2022-12-01T10:14:27.1089695Z dist init r=3, world=4
2022-12-01T10:14:27.1089875Z dist init r=0, world=4
2022-12-01T10:14:27.1090051Z dist init r=1, world=4
2022-12-01T10:14:27.1090245Z skip: Need at least 2 CUDA devices (1.417s)
2022-12-01T10:14:27.1090638Z   test_optim_state_dict_nested_state_dict_type_StateDictType_FULL_STATE_DICT_use_multiple_param_groups_True_rank0_only_True_use_diff_optim_inputs_False (__main__.TestFSDPOptimState)
2022-12-01T10:14:27.1091173Z Tests :meth:`full_optim_state_dict` and `sharded_optim_state_dict` ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8847
2022-12-01T10:14:27.1091625Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8848
2022-12-01T10:14:27.1091939Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8849
2022-12-01T10:14:27.1092264Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8850
2022-12-01T10:14:27.1092715Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:14:27.1093049Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:14:27.1093465Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:14:27.1093815Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:14:27.1094152Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:14:27.1094629Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:14:27.1094947Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:14:27.1095372Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:14:27.1095717Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:14:27.1096037Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:14:27.1096504Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:14:27.1096838Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:14:27.1097267Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:14:27.1097598Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:14:27.1098017Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-12-01T10:14:27.1098527Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:14:27.1099007Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:14:27.1099338Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:14:27.1099770Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:14:27.1100118Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:14:27.1100590Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:14:27.1101099Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:14:27.1101628Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:14:27.1102151Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:14:27.1102523Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:14:27.1102877Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:14:27.1103224Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:14:27.1103471Z dist init r=2, world=4
2022-12-01T10:14:27.1103653Z dist init r=3, world=4
2022-12-01T10:14:27.1103832Z dist init r=0, world=4
2022-12-01T10:14:27.1104167Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:14:27.1104418Z dist init r=1, world=4
2022-12-01T10:14:27.1104672Z skip: Need at least 2 CUDA devices (1.417s)
2022-12-01T10:14:27.1105062Z   test_optim_state_dict_nested_state_dict_type_StateDictType_FULL_STATE_DICT_use_multiple_param_groups_True_rank0_only_True_use_diff_optim_inputs_True (__main__.TestFSDPOptimState)
2022-12-01T10:14:27.1105541Z Tests :meth:`full_optim_state_dict` and `sharded_optim_state_dict` ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8903
2022-12-01T10:14:27.1105928Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8904
2022-12-01T10:14:27.1106254Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8905
2022-12-01T10:14:27.1106574Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8906
2022-12-01T10:14:27.1107009Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:14:27.1107340Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:14:27.1107770Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:14:27.1108104Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:14:27.1108439Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:14:27.1108907Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:14:27.1109234Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:14:27.1109648Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:14:27.1109994Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:14:27.1110333Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:14:27.1110799Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:14:27.1111113Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:14:27.1111537Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:14:27.1111879Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:14:27.1112201Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:14:27.1112663Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:14:27.1112993Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:14:27.1113418Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:14:27.1113747Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:14:27.1114081Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-12-01T10:14:27.1114573Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:14:27.1115095Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:14:27.1115601Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:14:27.1116114Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:14:27.1116503Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:14:27.1116876Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:14:27.1117254Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:14:27.1117595Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:14:27.1117853Z dist init r=1, world=4
2022-12-01T10:14:27.1118026Z dist init r=3, world=4
2022-12-01T10:14:27.1118209Z dist init r=2, world=4
2022-12-01T10:14:27.1118389Z dist init r=0, world=4
2022-12-01T10:14:27.1118584Z skip: Need at least 2 CUDA devices (1.417s)
2022-12-01T10:14:27.1118986Z   test_optim_state_dict_nested_state_dict_type_StateDictType_SHARDED_STATE_DICT_use_multiple_param_groups_False_rank0_only_False_use_diff_optim_inputs_False (__main__.TestFSDPOptimState)
2022-12-01T10:14:27.1119486Z Tests :meth:`full_optim_state_dict` and `sharded_optim_state_dict` ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8959
2022-12-01T10:14:27.1119880Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8960
2022-12-01T10:14:27.1120192Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8961
2022-12-01T10:14:27.1120509Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8962
2022-12-01T10:14:27.1120962Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:14:27.1121278Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:14:27.1121708Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:14:27.1122053Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:14:27.1122392Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:14:27.1122847Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:14:27.1123175Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:14:27.1123605Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:14:27.1123934Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:14:27.1124270Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:14:27.1124733Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:14:27.1125059Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:14:27.1125471Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:14:27.1125813Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:14:27.1126149Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:14:27.1126612Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:14:27.1126925Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:14:27.1127347Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:14:27.1127691Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:14:27.1128010Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-12-01T10:14:27.1128501Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:14:27.1129019Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:14:27.1129581Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:14:27.1130112Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:14:27.1130492Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:14:27.1130844Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:14:27.1131189Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:14:27.1131520Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:14:27.1131775Z dist init r=2, world=4
2022-12-01T10:14:27.1131961Z dist init r=3, world=4
2022-12-01T10:14:27.1132190Z dist init r=0, world=4
2022-12-01T10:14:27.1132368Z dist init r=1, world=4
2022-12-01T10:14:27.1132576Z skip: Need at least 2 CUDA devices (1.417s)
2022-12-01T10:14:27.1132984Z   test_optim_state_dict_nested_state_dict_type_StateDictType_SHARDED_STATE_DICT_use_multiple_param_groups_False_rank0_only_False_use_diff_optim_inputs_True (__main__.TestFSDPOptimState)
2022-12-01T10:14:27.1133489Z Tests :meth:`full_optim_state_dict` and `sharded_optim_state_dict` ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9015
2022-12-01T10:14:27.1133878Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9016
2022-12-01T10:14:27.1134203Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9017
2022-12-01T10:14:27.1134510Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9018
2022-12-01T10:14:27.1134960Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:14:27.1135292Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:14:27.1135722Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:14:27.1136056Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:14:27.1136397Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:14:27.1136869Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:14:27.1137185Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:14:27.1137613Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:14:27.1138043Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:14:27.1138385Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:14:27.1138848Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:14:27.1139181Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:14:27.1139609Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:14:27.1139953Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:14:27.1140278Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-12-01T10:14:27.1140895Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:14:27.1141228Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:14:27.1141640Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:14:27.1142053Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:14:27.1142394Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:14:27.1142934Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:14:27.1143443Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:14:27.1143964Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:14:27.1144480Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:14:27.1144870Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:14:27.1145208Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:14:27.1145549Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:14:27.1145895Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:14:27.1146139Z dist init r=2, world=4
2022-12-01T10:14:27.1146323Z dist init r=0, world=4
2022-12-01T10:14:27.1146501Z dist init r=3, world=4
2022-12-01T10:14:27.1146665Z dist init r=1, world=4
2022-12-01T10:14:27.1146870Z skip: Need at least 2 CUDA devices (1.418s)
2022-12-01T10:14:27.1147267Z   test_optim_state_dict_nested_state_dict_type_StateDictType_SHARDED_STATE_DICT_use_multiple_param_groups_False_rank0_only_True_use_diff_optim_inputs_False (__main__.TestFSDPOptimState)
2022-12-01T10:14:27.1147765Z Tests :meth:`full_optim_state_dict` and `sharded_optim_state_dict` ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9071
2022-12-01T10:14:27.1148138Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9072
2022-12-01T10:14:27.1148463Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9073
2022-12-01T10:14:27.1148791Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9074
2022-12-01T10:14:27.1149239Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:14:27.1149557Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:14:27.1149983Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:14:27.1150328Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:14:27.1150649Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:14:27.1151117Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:14:27.1151444Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:14:27.1151866Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:14:27.1152196Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:14:27.1152528Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:14:27.1152991Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:14:27.1153302Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:14:27.1153725Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:14:27.1154066Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:14:27.1154399Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-12-01T10:14:27.1154885Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:14:27.1155245Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:14:27.1155668Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:14:27.1156009Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:14:27.1156327Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:14:27.1156817Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:14:27.1157338Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:14:27.1157844Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:14:27.1158363Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:14:27.1158755Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:14:27.1159106Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:14:27.1159441Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:14:27.1159780Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:14:27.1160039Z dist init r=1, world=4
2022-12-01T10:14:27.1160220Z dist init r=2, world=4
2022-12-01T10:14:27.1160384Z dist init r=3, world=4
2022-12-01T10:14:27.1160561Z dist init r=0, world=4
2022-12-01T10:14:27.1160766Z skip: Need at least 2 CUDA devices (1.417s)
2022-12-01T10:14:27.1161149Z   test_optim_state_dict_nested_state_dict_type_StateDictType_SHARDED_STATE_DICT_use_multiple_param_groups_False_rank0_only_True_use_diff_optim_inputs_True (__main__.TestFSDPOptimState)
2022-12-01T10:14:27.1161648Z Tests :meth:`full_optim_state_dict` and `sharded_optim_state_dict` ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9127
2022-12-01T10:14:27.1162034Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9128
2022-12-01T10:14:27.1162348Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9129
2022-12-01T10:14:27.1162667Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9130
2022-12-01T10:14:27.1163118Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:14:27.1163450Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:14:27.1163881Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:14:27.1164230Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:14:27.1164569Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:14:27.1165038Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:14:27.1165352Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:14:27.1165776Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:14:27.1166119Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:14:27.1166439Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:14:27.1166902Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:14:27.1167266Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:14:27.1167724Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:14:27.1168052Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:14:27.1168390Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:14:27.1168862Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:14:27.1169175Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:14:27.1169603Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:14:27.1169945Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:14:27.1170283Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-12-01T10:14:27.1170766Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:14:27.1171290Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:14:27.1171807Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:14:27.1172323Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:14:27.1172692Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:14:27.1173040Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:14:27.1173384Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:14:27.1173728Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:14:27.1173975Z dist init r=0, world=4
2022-12-01T10:14:27.1174159Z dist init r=1, world=4
2022-12-01T10:14:27.1174336Z dist init r=3, world=4
2022-12-01T10:14:27.1174501Z dist init r=2, world=4
2022-12-01T10:14:27.1174707Z skip: Need at least 2 CUDA devices (1.417s)
2022-12-01T10:14:27.1175102Z   test_optim_state_dict_nested_state_dict_type_StateDictType_SHARDED_STATE_DICT_use_multiple_param_groups_True_rank0_only_False_use_diff_optim_inputs_False (__main__.TestFSDPOptimState)
2022-12-01T10:14:27.1175580Z Tests :meth:`full_optim_state_dict` and `sharded_optim_state_dict` ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9183
2022-12-01T10:14:27.1175967Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9184
2022-12-01T10:14:27.1176293Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9185
2022-12-01T10:14:27.1176618Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9186
2022-12-01T10:14:27.1177060Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:14:27.1177389Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:14:27.1177820Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:14:27.1178264Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:14:27.1178589Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:14:27.1179064Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:14:27.1179392Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:14:27.1179847Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:14:27.1180242Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:14:27.1180718Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:14:27.1181191Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:14:27.1181503Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:14:27.1181932Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:14:27.1182278Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:14:27.1182600Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-12-01T10:14:27.1183074Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:14:27.1183403Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:14:27.1183835Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:14:27.1184164Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:14:27.1184500Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:14:27.1184997Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:14:27.1185514Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:14:27.1186015Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:14:27.1186532Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:14:27.1186919Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:14:27.1187253Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:14:27.1187594Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:14:27.1187929Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:14:27.1188188Z dist init r=2, world=4
2022-12-01T10:14:27.1188357Z dist init r=1, world=4
2022-12-01T10:14:27.1188536Z dist init r=0, world=4
2022-12-01T10:14:27.1188712Z dist init r=3, world=4
2022-12-01T10:14:27.1188905Z skip: Need at least 2 CUDA devices (1.418s)
2022-12-01T10:14:27.1189301Z   test_optim_state_dict_nested_state_dict_type_StateDictType_SHARDED_STATE_DICT_use_multiple_param_groups_True_rank0_only_False_use_diff_optim_inputs_True (__main__.TestFSDPOptimState)
2022-12-01T10:14:27.1189791Z Tests :meth:`full_optim_state_dict` and `sharded_optim_state_dict` ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9239
2022-12-01T10:14:27.1190183Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9240
2022-12-01T10:14:27.1190493Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9241
2022-12-01T10:14:27.1190807Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9242
2022-12-01T10:14:27.1191257Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:14:27.1191575Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:14:27.1192007Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:14:27.1192354Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:14:27.1192753Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:14:27.1193252Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:14:27.1193583Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:14:27.1194012Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:14:27.1194358Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:14:27.1194684Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:14:27.1195152Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:14:27.1195481Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:14:27.1195897Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:14:27.1196247Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:14:27.1196582Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-12-01T10:14:27.1197045Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:14:27.1197355Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:14:27.1197777Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:14:27.1198117Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:14:27.1198437Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:14:27.1198929Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:14:27.1199456Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:14:27.1199973Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:14:27.1200465Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:14:27.1200854Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:14:27.1201202Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:14:27.1201546Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:14:27.1201876Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:14:27.1202130Z dist init r=1, world=4
2022-12-01T10:14:27.1202315Z dist init r=2, world=4
2022-12-01T10:14:27.1202481Z dist init r=3, world=4
2022-12-01T10:14:27.1202660Z dist init r=0, world=4
2022-12-01T10:14:27.1202866Z skip: Need at least 2 CUDA devices (1.417s)
2022-12-01T10:14:27.1203250Z   test_optim_state_dict_nested_state_dict_type_StateDictType_SHARDED_STATE_DICT_use_multiple_param_groups_True_rank0_only_True_use_diff_optim_inputs_False (__main__.TestFSDPOptimState)
2022-12-01T10:14:27.1203745Z Tests :meth:`full_optim_state_dict` and `sharded_optim_state_dict` ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9295
2022-12-01T10:14:27.1204138Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9296
2022-12-01T10:14:27.1204466Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9297
2022-12-01T10:14:27.1204777Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9298
2022-12-01T10:14:27.1205264Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:14:27.1205626Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:14:27.1206055Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:14:27.1206387Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:14:27.1206724Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:14:27.1207196Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:14:27.1207511Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:14:27.1207938Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:14:27.1208289Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:14:27.1208627Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:14:27.1209079Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:14:27.1209409Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:14:27.1209836Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:14:27.1210181Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:14:27.1210504Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:14:27.1210968Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:14:27.1211295Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:14:27.1211707Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:14:27.1212055Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:14:27.1212390Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-12-01T10:14:27.1212883Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:14:27.1213389Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:14:27.1213901Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:14:27.1214414Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:14:27.1214797Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:14:27.1215133Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:14:27.1215472Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:14:27.1215817Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:14:27.1216064Z dist init r=1, world=4
2022-12-01T10:14:27.1216248Z dist init r=0, world=4
2022-12-01T10:14:27.1216427Z dist init r=2, world=4
2022-12-01T10:14:27.1216590Z dist init r=3, world=4
2022-12-01T10:14:27.1216796Z skip: Need at least 2 CUDA devices (1.417s)
2022-12-01T10:14:27.1217193Z   test_optim_state_dict_nested_state_dict_type_StateDictType_SHARDED_STATE_DICT_use_multiple_param_groups_True_rank0_only_True_use_diff_optim_inputs_True (__main__.TestFSDPOptimState)
2022-12-01T10:14:27.1217719Z Tests :meth:`full_optim_state_dict` and `sharded_optim_state_dict` ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9351
2022-12-01T10:14:27.1218208Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9352
2022-12-01T10:14:27.1218539Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9353
2022-12-01T10:14:27.1218863Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9354
2022-12-01T10:14:27.1219322Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:14:27.1219643Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:14:27.1220072Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:14:27.1220574Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:14:27.1221013Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:14:27.1221699Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:14:27.1222130Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:14:27.1222564Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:14:27.1222898Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:14:27.1223242Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:14:27.1223714Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:14:27.1224045Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:14:27.1224460Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:14:27.1224814Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:14:27.1225155Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-12-01T10:14:27.1225610Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:14:27.1225943Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:14:27.1226370Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:14:27.1226712Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:14:27.1227032Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:14:27.1227530Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:14:27.1228053Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:14:27.1228576Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:14:27.1229076Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:14:27.1229461Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:14:27.1229809Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:14:27.1230138Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:14:27.1230482Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:14:27.1230736Z dist init r=0, world=4
2022-12-01T10:14:27.1230917Z dist init r=1, world=4
2022-12-01T10:14:27.1231172Z dist init r=2, world=4
2022-12-01T10:14:27.1231400Z dist init r=3, world=4
2022-12-01T10:14:27.1231610Z skip: Need at least 2 CUDA devices (1.417s)
2022-12-01T10:14:27.1231953Z   test_rekey_optim_state_dict_to_ids_state_dict_type_StateDictType_FULL_STATE_DICT_use_multiple_param_groups_False (__main__.TestFSDPOptimState)
2022-12-01T10:14:27.1232410Z Tests :meth:`rekey_optim_state_dict` with the new keys being ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9407
2022-12-01T10:14:27.1232796Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9408
2022-12-01T10:14:27.1233109Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9409
2022-12-01T10:14:27.1233434Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9410
2022-12-01T10:14:27.1233887Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:14:27.1234222Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:14:27.1234638Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:14:27.1234984Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:14:27.1235318Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:14:27.1235827Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:14:27.1236142Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:14:27.1236568Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:14:27.1236912Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:14:27.1237235Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:14:27.1237701Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:14:27.1238030Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:14:27.1238454Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:14:27.1238779Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:14:27.1239113Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:14:27.1239578Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:14:27.1239891Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:14:27.1240319Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:14:27.1240666Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:14:27.1241003Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-12-01T10:14:27.1241481Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:14:27.1241995Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:14:27.1242511Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:14:27.1243020Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:14:27.1243394Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:14:27.1243781Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:14:27.1244161Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:14:27.1244505Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:14:27.1244751Z dist init r=2, world=4
2022-12-01T10:14:27.1244933Z dist init r=0, world=4
2022-12-01T10:14:27.1245112Z dist init r=3, world=4
2022-12-01T10:14:27.1245277Z dist init r=1, world=4
2022-12-01T10:14:27.1245483Z skip: Need at least 2 CUDA devices (1.417s)
2022-12-01T10:14:27.1245839Z   test_rekey_optim_state_dict_to_ids_state_dict_type_StateDictType_FULL_STATE_DICT_use_multiple_param_groups_True (__main__.TestFSDPOptimState)
2022-12-01T10:14:27.1246282Z Tests :meth:`rekey_optim_state_dict` with the new keys being ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9463
2022-12-01T10:14:27.1246672Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9464
2022-12-01T10:14:27.1247000Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9465
2022-12-01T10:14:27.1247323Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9466
2022-12-01T10:14:27.1247764Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:14:27.1248097Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:14:27.1248527Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:14:27.1248863Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:14:27.1249205Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:14:27.1249671Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:14:27.1250002Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:14:27.1250420Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:14:27.1250765Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:14:27.1251102Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:14:27.1251571Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:14:27.1251887Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:14:27.1252320Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:14:27.1252665Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:14:27.1252988Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-12-01T10:14:27.1253460Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:14:27.1253798Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:14:27.1254224Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:14:27.1254554Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:14:27.1254888Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:14:27.1255380Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:14:27.1255896Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:14:27.1256438Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:14:27.1257000Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:14:27.1257387Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:14:27.1257724Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:14:27.1258159Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:14:27.1258501Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:14:27.1258764Z dist init r=2, world=4
2022-12-01T10:14:27.1258934Z dist init r=1, world=4
2022-12-01T10:14:27.1259114Z dist init r=0, world=4
2022-12-01T10:14:27.1259296Z dist init r=3, world=4
2022-12-01T10:14:27.1259492Z skip: Need at least 2 CUDA devices (1.417s)
2022-12-01T10:14:27.1259853Z   test_rekey_optim_state_dict_to_ids_state_dict_type_StateDictType_SHARDED_STATE_DICT_use_multiple_param_groups_False (__main__.TestFSDPOptimState)
2022-12-01T10:14:27.1260313Z Tests :meth:`rekey_optim_state_dict` with the new keys being ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9519
2022-12-01T10:14:27.1260839Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9520
2022-12-01T10:14:27.1261156Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9521
2022-12-01T10:14:27.1261478Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9522
2022-12-01T10:14:27.1261937Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:14:27.1262257Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:14:27.1262693Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:14:27.1263042Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:14:27.1263382Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:14:27.1263832Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:14:27.1264158Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:14:27.1264590Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:14:27.1264922Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:14:27.1265258Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:14:27.1265727Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:14:27.1266055Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:14:27.1266470Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:14:27.1266811Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:14:27.1267145Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-12-01T10:14:27.1267638Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:14:27.1268121Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:14:27.1268506Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:14:27.1268934Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:14:27.1269332Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:14:27.1269734Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:14:27.1270229Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:14:27.1270747Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:14:27.1271252Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:14:27.1271677Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:14:27.1272024Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:14:27.1272371Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:14:27.1272703Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:14:27.1272964Z dist init r=1, world=4
2022-12-01T10:14:27.1273150Z dist init r=0, world=4
2022-12-01T10:14:27.1273316Z dist init r=2, world=4
2022-12-01T10:14:27.1273492Z dist init r=3, world=4
2022-12-01T10:14:27.1273698Z skip: Need at least 2 CUDA devices (1.417s)
2022-12-01T10:14:27.1274043Z   test_rekey_optim_state_dict_to_ids_state_dict_type_StateDictType_SHARDED_STATE_DICT_use_multiple_param_groups_True (__main__.TestFSDPOptimState)
2022-12-01T10:14:27.1274501Z Tests :meth:`rekey_optim_state_dict` with the new keys being ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9575
2022-12-01T10:14:27.1274889Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9576
2022-12-01T10:14:27.1275215Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9577
2022-12-01T10:14:27.1275525Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9578
2022-12-01T10:14:27.1275969Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:14:27.1276302Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:14:27.1276718Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:14:27.1277064Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:14:27.1277401Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:14:27.1277866Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:14:27.1278180Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:14:27.1278607Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:14:27.1278952Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:14:27.1279288Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:14:27.1279744Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:14:27.1280081Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:14:27.1280504Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:14:27.1280832Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:14:27.1281166Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:14:27.1281659Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:14:27.1282195Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:14:27.1282540Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:14:27.1282964Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:14:27.1283308Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:14:27.1283644Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-12-01T10:14:27.1284120Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:14:27.1284635Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:14:27.1285145Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:14:27.1285520Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:14:27.1285873Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:14:27.1286216Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:14:27.1286560Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:14:27.1286805Z dist init r=1, world=4
2022-12-01T10:14:27.1286986Z dist init r=2, world=4
2022-12-01T10:14:27.1287167Z dist init r=0, world=4
2022-12-01T10:14:27.1287388Z dist init r=3, world=4
2022-12-01T10:14:27.1287592Z skip: Need at least 2 CUDA devices (1.416s)
2022-12-01T10:14:27.1287863Z   test_rekey_optim_state_dict_to_names (__main__.TestFSDPOptimState)
2022-12-01T10:14:27.1288233Z Tests :meth:`rekey_optim_state_dict` with the new keys being ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9631
2022-12-01T10:14:27.1288624Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9632
2022-12-01T10:14:27.1288950Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9633
2022-12-01T10:14:27.1289271Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9634
2022-12-01T10:14:27.1289710Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:14:27.1290042Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:14:27.1290501Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:14:27.1290848Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:14:27.1291173Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:14:27.1291644Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:14:27.1291972Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:14:27.1292387Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:14:27.1292729Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:14:27.1293068Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:14:27.1293531Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:14:27.1293844Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:14:27.1294268Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:14:27.1294643Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:14:27.1294965Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:14:27.1295464Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:14:27.1295793Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:14:27.1296218Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:14:27.1296549Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:14:27.1296884Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-12-01T10:14:27.1297376Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:14:27.1297966Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:14:27.1298482Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:14:27.1299002Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:14:27.1299395Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:14:27.1299745Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:14:27.1300077Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:14:27.1300554Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:14:27.1300823Z dist init r=3, world=4
2022-12-01T10:14:27.1300995Z dist init r=1, world=4
2022-12-01T10:14:27.1301180Z dist init r=0, world=4
2022-12-01T10:14:27.1301366Z dist init r=2, world=4
2022-12-01T10:14:27.1301564Z skip: Need at least 2 CUDA devices (1.416s)
2022-12-01T10:14:27.1301862Z   test_scatter_full_optim_state_dict_nested_halve_world_size (__main__.TestFSDPOptimState)
2022-12-01T10:14:27.1302390Z Tests :meth:`scatter_full_optim_state_dict` for a non-FSDP-root ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9687
2022-12-01T10:14:27.1302791Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9688
2022-12-01T10:14:27.1303105Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9689
2022-12-01T10:14:27.1303423Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9690
2022-12-01T10:14:27.1303873Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:14:27.1304191Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:14:27.1304624Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:14:27.1304975Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:14:27.1305314Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:14:27.1305768Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:14:27.1306095Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:14:27.1306524Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:14:27.1306859Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:14:27.1307196Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:14:27.1307724Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:14:27.1308096Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:14:27.1308510Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:14:27.1308856Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:14:27.1309193Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-12-01T10:14:27.1309688Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:14:27.1310171Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:14:27.1310499Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:14:27.1310927Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:14:27.1311257Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:14:27.1311591Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:14:27.1312081Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:14:27.1312600Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:14:27.1313103Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:14:27.1313491Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:14:27.1313838Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:14:27.1314183Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:14:27.1314333Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:14:27.1314410Z dist init r=2, world=4
2022-12-01T10:14:27.1314483Z dist init r=0, world=4
2022-12-01T10:14:27.1314554Z dist init r=1, world=4
2022-12-01T10:14:27.1314626Z dist init r=3, world=4
2022-12-01T10:14:27.1314729Z skip: Need at least 2 CUDA devices (1.417s)
2022-12-01T10:14:27.1314959Z   test_scatter_full_optim_state_dict_nested_use_multiple_param_groups_False_wrap_alt_False_use_diff_optim_inputs_False (__main__.TestFSDPOptimState)
2022-12-01T10:14:27.1315288Z Tests :meth:`scatter_full_optim_state_dict` for a non-FSDP-root ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9743
2022-12-01T10:14:27.1315429Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9744
2022-12-01T10:14:27.1315581Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9745
2022-12-01T10:14:27.1315732Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9746
2022-12-01T10:14:27.1316006Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:14:27.1316131Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:14:27.1316413Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:14:27.1316550Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:14:27.1316725Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:14:27.1316982Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:14:27.1317106Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:14:27.1317416Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:14:27.1317586Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:14:27.1317761Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:14:27.1318029Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:14:27.1318151Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:14:27.1318426Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:14:27.1318558Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:14:27.1318719Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:14:27.1318985Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:14:27.1319108Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:14:27.1319384Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:14:27.1319516Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:14:27.1319687Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-12-01T10:14:27.1319983Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:14:27.1320279Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:14:27.1320577Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:14:27.1320861Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:14:27.1321030Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:14:27.1321194Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:14:27.1321358Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:14:27.1321521Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:14:27.1321598Z dist init r=3, world=4
2022-12-01T10:14:27.1321672Z dist init r=2, world=4
2022-12-01T10:14:27.1321744Z dist init r=1, world=4
2022-12-01T10:14:27.1321804Z dist init r=0, world=4
2022-12-01T10:14:27.1321908Z skip: Need at least 2 CUDA devices (1.417s)
2022-12-01T10:14:27.1322135Z   test_scatter_full_optim_state_dict_nested_use_multiple_param_groups_False_wrap_alt_False_use_diff_optim_inputs_True (__main__.TestFSDPOptimState)
2022-12-01T10:14:27.1322468Z Tests :meth:`scatter_full_optim_state_dict` for a non-FSDP-root ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9799
2022-12-01T10:14:27.1322624Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9800
2022-12-01T10:14:27.1322774Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9801
2022-12-01T10:14:27.1322919Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9802
2022-12-01T10:14:27.1323193Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:14:27.1323304Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:14:27.1323581Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:14:27.1323754Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:14:27.1323935Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:14:27.1324238Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:14:27.1324361Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:14:27.1324636Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:14:27.1324770Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:14:27.1324947Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:14:27.1325205Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:14:27.1325329Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:14:27.1325609Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:14:27.1325745Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:14:27.1325921Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-12-01T10:14:27.1326218Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:14:27.1326512Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:14:27.1326784Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:14:27.1326895Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:14:27.1327176Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:14:27.1327310Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:14:27.1327487Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:14:27.1327783Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:14:27.1328078Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:14:27.1328243Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:14:27.1328406Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:14:27.1328562Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:14:27.1328707Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:14:27.1328789Z dist init r=1, world=4
2022-12-01T10:14:27.1328864Z dist init r=0, world=4
2022-12-01T10:14:27.1328936Z dist init r=2, world=4
2022-12-01T10:14:27.1329008Z dist init r=3, world=4
2022-12-01T10:14:27.1329113Z skip: Need at least 2 CUDA devices (1.417s)
2022-12-01T10:14:27.1329342Z   test_scatter_full_optim_state_dict_nested_use_multiple_param_groups_False_wrap_alt_True_use_diff_optim_inputs_False (__main__.TestFSDPOptimState)
2022-12-01T10:14:27.1329675Z Tests :meth:`scatter_full_optim_state_dict` for a non-FSDP-root ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9855
2022-12-01T10:14:27.1329817Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9856
2022-12-01T10:14:27.1329968Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9857
2022-12-01T10:14:27.1330112Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9858
2022-12-01T10:14:27.1330421Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:14:27.1330574Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:14:27.1330855Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:14:27.1330992Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:14:27.1331169Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:14:27.1331423Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:14:27.1331548Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:14:27.1331824Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:14:27.1331959Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:14:27.1332135Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:14:27.1332434Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:14:27.1332704Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:14:27.1332827Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:14:27.1333106Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:14:27.1333226Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:14:27.1333402Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-12-01T10:14:27.1333675Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:14:27.1333799Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:14:27.1334083Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:14:27.1334218Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:14:27.1334398Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:14:27.1334697Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:14:27.1334991Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:14:27.1335273Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:14:27.1335441Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:14:27.1335606Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:14:27.1335763Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:14:27.1335919Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:14:27.1335993Z dist init r=0, world=4
2022-12-01T10:14:27.1336066Z dist init r=1, world=4
2022-12-01T10:14:27.1336137Z dist init r=3, world=4
2022-12-01T10:14:27.1336197Z dist init r=2, world=4
2022-12-01T10:14:27.1336303Z skip: Need at least 2 CUDA devices (1.417s)
2022-12-01T10:14:27.1336528Z   test_scatter_full_optim_state_dict_nested_use_multiple_param_groups_False_wrap_alt_True_use_diff_optim_inputs_True (__main__.TestFSDPOptimState)
2022-12-01T10:14:27.1336889Z Tests :meth:`scatter_full_optim_state_dict` for a non-FSDP-root ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9911
2022-12-01T10:14:27.1337047Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9912
2022-12-01T10:14:27.1337248Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9913
2022-12-01T10:14:27.1337402Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9914
2022-12-01T10:14:27.1337672Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:14:27.1337782Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:14:27.1338162Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:14:27.1338309Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:14:27.1338484Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:14:27.1338760Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:14:27.1338888Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:14:27.1339165Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:14:27.1339301Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:14:27.1339474Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:14:27.1339726Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:14:27.1339851Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:14:27.1340126Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:14:27.1340263Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:14:27.1340575Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-12-01T10:14:27.1340850Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:14:27.1340971Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:14:27.1341245Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:14:27.1341364Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:14:27.1341538Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:14:27.1341833Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:14:27.1342131Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:14:27.1342431Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:14:27.1342720Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:14:27.1342888Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:14:27.1343052Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:14:27.1343215Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:14:27.1343364Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:14:27.1343442Z dist init r=3, world=4
2022-12-01T10:14:27.1343515Z dist init r=2, world=4
2022-12-01T10:14:27.1343587Z dist init r=0, world=4
2022-12-01T10:14:27.1343721Z dist init r=1, world=4
2022-12-01T10:14:27.1343831Z skip: Need at least 2 CUDA devices (1.417s)
2022-12-01T10:14:27.1344101Z   test_scatter_full_optim_state_dict_nested_use_multiple_param_groups_True_wrap_alt_False_use_diff_optim_inputs_False (__main__.TestFSDPOptimState)
2022-12-01T10:14:27.1344431Z Tests :meth:`scatter_full_optim_state_dict` for a non-FSDP-root ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9967
2022-12-01T10:14:27.1344573Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9968
2022-12-01T10:14:27.1344724Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9969
2022-12-01T10:14:27.1344869Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9970
2022-12-01T10:14:27.1345147Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:14:27.1345272Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:14:27.1345557Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:14:27.1345697Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:14:27.1345876Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:14:27.1346148Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:14:27.1346256Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:14:27.1346532Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:14:27.1346665Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:14:27.1346840Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:14:27.1347112Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:14:27.1347235Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:14:27.1347512Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:14:27.1347644Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:14:27.1347804Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-12-01T10:14:27.1348068Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:14:27.1348188Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:14:27.1348462Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:14:27.1348596Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:14:27.1348771Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:14:27.1349071Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:14:27.1349367Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:14:27.1349663Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:14:27.1349943Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:14:27.1350107Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:14:27.1350267Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:14:27.1350459Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:14:27.1350647Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:14:27.1350724Z dist init r=2, world=4
2022-12-01T10:14:27.1350797Z dist init r=0, world=4
2022-12-01T10:14:27.1350871Z dist init r=1, world=4
2022-12-01T10:14:27.1350930Z dist init r=3, world=4
2022-12-01T10:14:27.1351035Z skip: Need at least 2 CUDA devices (1.417s)
2022-12-01T10:14:27.1351259Z   test_scatter_full_optim_state_dict_nested_use_multiple_param_groups_True_wrap_alt_False_use_diff_optim_inputs_True (__main__.TestFSDPOptimState)
2022-12-01T10:14:27.1351589Z Tests :meth:`scatter_full_optim_state_dict` for a non-FSDP-root ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10023
2022-12-01T10:14:27.1351747Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10024
2022-12-01T10:14:27.1351899Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10025
2022-12-01T10:14:27.1352051Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10026
2022-12-01T10:14:27.1352324Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:14:27.1352433Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:14:27.1352712Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:14:27.1352845Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:14:27.1353022Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:14:27.1353289Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:14:27.1353413Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:14:27.1353691Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:14:27.1353826Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:14:27.1354000Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:14:27.1354260Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:14:27.1354384Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:14:27.1354660Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:14:27.1354794Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:14:27.1354972Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-12-01T10:14:27.1355241Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:14:27.1355367Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:14:27.1355643Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:14:27.1355763Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:14:27.1355934Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:14:27.1356231Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:14:27.1356525Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:14:27.1356853Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:14:27.1357151Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:14:27.1357348Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:14:27.1357513Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:14:27.1357672Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:14:27.1357830Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:14:27.1357893Z dist init r=2, world=4
2022-12-01T10:14:27.1357967Z dist init r=1, world=4
2022-12-01T10:14:27.1358040Z dist init r=0, world=4
2022-12-01T10:14:27.1358113Z dist init r=3, world=4
2022-12-01T10:14:27.1358219Z skip: Need at least 2 CUDA devices (1.417s)
2022-12-01T10:14:27.1358447Z   test_scatter_full_optim_state_dict_nested_use_multiple_param_groups_True_wrap_alt_True_use_diff_optim_inputs_False (__main__.TestFSDPOptimState)
2022-12-01T10:14:27.1358780Z Tests :meth:`scatter_full_optim_state_dict` for a non-FSDP-root ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10079
2022-12-01T10:14:27.1358926Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10080
2022-12-01T10:14:27.1359081Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10081
2022-12-01T10:14:27.1359230Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10082
2022-12-01T10:14:27.1359506Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:14:27.1359629Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:14:27.1359908Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:14:27.1360047Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:14:27.1360225Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:14:27.1360492Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:14:27.1360602Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:14:27.1360877Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:14:27.1361010Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:14:27.1361186Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:14:27.1361450Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:14:27.1361574Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:14:27.1361846Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:14:27.1361980Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:14:27.1362139Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:14:27.1362401Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:14:27.1362523Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:14:27.1362796Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:14:27.1362927Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:14:27.1363099Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-12-01T10:14:27.1363427Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:14:27.1363754Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:14:27.1364049Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:14:27.1364328Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:14:27.1364492Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:14:27.1364653Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:14:27.1364811Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:14:27.1364972Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:14:27.1365051Z dist init r=2, world=4
2022-12-01T10:14:27.1365123Z dist init r=3, world=4
2022-12-01T10:14:27.1365195Z dist init r=0, world=4
2022-12-01T10:14:27.1365254Z dist init r=1, world=4
2022-12-01T10:14:27.1365360Z skip: Need at least 2 CUDA devices (1.417s)
2022-12-01T10:14:27.1365585Z   test_scatter_full_optim_state_dict_nested_use_multiple_param_groups_True_wrap_alt_True_use_diff_optim_inputs_True (__main__.TestFSDPOptimState)
2022-12-01T10:14:27.1365916Z Tests :meth:`scatter_full_optim_state_dict` for a non-FSDP-root ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10135
2022-12-01T10:14:27.1366073Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10136
2022-12-01T10:14:27.1366223Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10137
2022-12-01T10:14:27.1366368Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10138
2022-12-01T10:14:27.1366650Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:14:27.1366811Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:14:27.1367257Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:14:27.1367467Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:14:27.1367742Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:14:27.1368216Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:14:27.1368410Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:14:27.1368907Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:14:27.1369157Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:14:27.1369473Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:14:27.1369934Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:14:27.1370138Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:14:27.1370612Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:14:27.1370838Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:14:27.1371146Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-12-01T10:14:27.1371607Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:14:27.1371951Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:14:27.1372464Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:14:27.1372779Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:14:27.1373067Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:14:27.1373595Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:14:27.1374073Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:14:27.1374400Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:14:27.1374700Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:14:27.1374873Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:14:27.1375044Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:14:27.1375206Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:14:27.1375365Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:14:27.1375429Z dist init r=3, world=4
2022-12-01T10:14:27.1375503Z dist init r=1, world=4
2022-12-01T10:14:27.1375581Z dist init r=0, world=4
2022-12-01T10:14:27.1375654Z dist init r=2, world=4
2022-12-01T10:14:27.1375762Z skip: Need at least 2 CUDA devices (1.417s)
2022-12-01T10:14:27.1375917Z   test_scatter_full_optim_state_dict_transformer (__main__.TestFSDPOptimState)
2022-12-01T10:14:27.1376243Z Tests :meth:`scatter_full_optim_state_dict` for an FSDP-root ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10191
2022-12-01T10:14:27.1376386Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10192
2022-12-01T10:14:27.1376538Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10193
2022-12-01T10:14:27.1376684Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10194
2022-12-01T10:14:27.1376962Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:14:27.1377087Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:14:27.1377372Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:14:27.1377508Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:14:27.1377686Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:14:27.1378038Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:14:27.1378153Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:14:27.1378432Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:14:27.1378567Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:14:27.1378746Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:14:27.1379014Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:14:27.1379137Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:14:27.1379415Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:14:27.1379550Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:14:27.1379769Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-12-01T10:14:27.1380074Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:14:27.1380196Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:14:27.1380618Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:14:27.1380756Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:14:27.1380930Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:14:27.1381235Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:14:27.1381539Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:14:27.1381840Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:14:27.1382125Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:14:27.1382294Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:14:27.1382459Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:14:27.1382619Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:14:27.1382779Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:14:27.1382858Z dist init r=2, world=4
2022-12-01T10:14:27.1382934Z dist init r=0, world=4
2022-12-01T10:14:27.1383009Z dist init r=3, world=4
2022-12-01T10:14:27.1383069Z dist init r=1, world=4
2022-12-01T10:14:27.1383179Z skip: Need at least 2 CUDA devices (1.416s)
2022-12-01T10:14:27.1383346Z   test_shard_full_optim_state_dict_nested_halve_world_size (__main__.TestFSDPOptimState)
2022-12-01T10:14:27.1383688Z Tests :meth:`shard_full_optim_state_dict` for a non-FSDP-root model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10247
2022-12-01T10:14:27.1383847Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10248
2022-12-01T10:14:27.1383997Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10249
2022-12-01T10:14:27.1384146Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10250
2022-12-01T10:14:27.1384425Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:14:27.1384538Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:14:27.1384823Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:14:27.1384964Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:14:27.1385142Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:14:27.1385415Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:14:27.1385539Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:14:27.1385819Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:14:27.1385954Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:14:27.1386131Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:14:27.1386451Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:14:27.1386579Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:14:27.1386920Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:14:27.1387056Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:14:27.1387231Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:14:27.1387498Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:14:27.1387620Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:14:27.1387893Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:14:27.1388013Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:14:27.1388216Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-12-01T10:14:27.1388518Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:14:27.1388814Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:14:27.1389111Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:14:27.1389402Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:14:27.1389568Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:14:27.1389734Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:14:27.1389896Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:14:27.1390043Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:14:27.1390120Z dist init r=0, world=4
2022-12-01T10:14:27.1390194Z dist init r=3, world=4
2022-12-01T10:14:27.1390268Z dist init r=2, world=4
2022-12-01T10:14:27.1390340Z dist init r=1, world=4
2022-12-01T10:14:27.1390446Z skip: Need at least 2 CUDA devices (1.419s)
2022-12-01T10:14:27.1390672Z   test_shard_full_optim_state_dict_nested_use_multiple_param_groups_False_wrap_alt_False_use_diff_optim_inputs_False (__main__.TestFSDPOptimState)
2022-12-01T10:14:27.1391011Z Tests :meth:`shard_full_optim_state_dict` for a non-FSDP-root model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10303
2022-12-01T10:14:27.1391153Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10304
2022-12-01T10:14:27.1391302Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10305
2022-12-01T10:14:27.1391451Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10306
2022-12-01T10:14:27.1391730Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:14:27.1391854Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:14:27.1392135Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:14:27.1392274Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:14:27.1392452Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:14:27.1392708Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:14:27.1392830Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:14:27.1393140Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:14:27.1393305Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:14:27.1393481Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:14:27.1393750Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:14:27.1393872Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:14:27.1394145Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:14:27.1394278Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:14:27.1394439Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-12-01T10:14:27.1394737Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:14:27.1395034Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:14:27.1395329Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:14:27.1395600Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:14:27.1395723Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:14:27.1396003Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:14:27.1396140Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:14:27.1396320Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:14:27.1396606Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:14:27.1396773Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:14:27.1396937Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:14:27.1397097Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:14:27.1397257Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:14:27.1397334Z dist init r=3, world=4
2022-12-01T10:14:27.1397406Z dist init r=2, world=4
2022-12-01T10:14:27.1397478Z dist init r=1, world=4
2022-12-01T10:14:27.1397536Z dist init r=0, world=4
2022-12-01T10:14:27.1397641Z skip: Need at least 2 CUDA devices (1.417s)
2022-12-01T10:14:27.1397867Z   test_shard_full_optim_state_dict_nested_use_multiple_param_groups_False_wrap_alt_False_use_diff_optim_inputs_True (__main__.TestFSDPOptimState)
2022-12-01T10:14:27.1398205Z Tests :meth:`shard_full_optim_state_dict` for a non-FSDP-root model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10359
2022-12-01T10:14:27.1398363Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10360
2022-12-01T10:14:27.1398514Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10361
2022-12-01T10:14:27.1398659Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10362
2022-12-01T10:14:27.1398936Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:14:27.1399047Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:14:27.1399326Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:14:27.1399461Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:14:27.1399673Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:14:27.1399971Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:14:27.1400096Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:14:27.1400370Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:14:27.1400504Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:14:27.1400678Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:14:27.1400932Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:14:27.1401053Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:14:27.1401328Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:14:27.1401466Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:14:27.1401644Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-12-01T10:14:27.1401940Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:14:27.1402207Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:14:27.1402329Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:14:27.1402604Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:14:27.1402725Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:14:27.1402902Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:14:27.1403199Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:14:27.1403492Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:14:27.1403784Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:14:27.1403949Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:14:27.1404115Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:14:27.1404272Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:14:27.1404419Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:14:27.1404497Z dist init r=2, world=4
2022-12-01T10:14:27.1404570Z dist init r=1, world=4
2022-12-01T10:14:27.1404644Z dist init r=3, world=4
2022-12-01T10:14:27.1404716Z dist init r=0, world=4
2022-12-01T10:14:27.1404819Z skip: Need at least 2 CUDA devices (1.417s)
2022-12-01T10:14:27.1405041Z   test_shard_full_optim_state_dict_nested_use_multiple_param_groups_False_wrap_alt_True_use_diff_optim_inputs_False (__main__.TestFSDPOptimState)
2022-12-01T10:14:27.1405372Z Tests :meth:`shard_full_optim_state_dict` for a non-FSDP-root model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10415
2022-12-01T10:14:27.1405517Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10416
2022-12-01T10:14:27.1405665Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10417
2022-12-01T10:14:27.1405811Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10418
2022-12-01T10:14:27.1406115Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:14:27.1406270Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:14:27.1406553Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:14:27.1406691Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:14:27.1406866Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:14:27.1407136Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:14:27.1407246Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:14:27.1407523Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:14:27.1407658Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:14:27.1407833Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:14:27.1408102Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:14:27.1408224Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:14:27.1408498Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:14:27.1408632Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:14:27.1408792Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-12-01T10:14:27.1409093Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:14:27.1409364Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:14:27.1409488Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:14:27.1409769Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:14:27.1409903Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:14:27.1410080Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:14:27.1410375Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:14:27.1410673Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:14:27.1410954Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:14:27.1411121Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:14:27.1411286Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:14:27.1411445Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:14:27.1411601Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:14:27.1411677Z dist init r=2, world=4
2022-12-01T10:14:27.1411750Z dist init r=0, world=4
2022-12-01T10:14:27.1411822Z dist init r=1, world=4
2022-12-01T10:14:27.1411881Z dist init r=3, world=4
2022-12-01T10:14:27.1411986Z skip: Need at least 2 CUDA devices (1.418s)
2022-12-01T10:14:27.1412208Z   test_shard_full_optim_state_dict_nested_use_multiple_param_groups_False_wrap_alt_True_use_diff_optim_inputs_True (__main__.TestFSDPOptimState)
2022-12-01T10:14:27.1412574Z Tests :meth:`shard_full_optim_state_dict` for a non-FSDP-root model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10471
2022-12-01T10:14:27.1412731Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10472
2022-12-01T10:14:27.1412912Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10473
2022-12-01T10:14:27.1413058Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10474
2022-12-01T10:14:27.1413330Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:14:27.1413440Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:14:27.1413718Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:14:27.1413854Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:14:27.1414032Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:14:27.1414303Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:14:27.1414427Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:14:27.1414707Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:14:27.1414841Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:14:27.1415018Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:14:27.1415272Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:14:27.1415397Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:14:27.1415671Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:14:27.1415808Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:14:27.1415983Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-12-01T10:14:27.1416252Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:14:27.1416373Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:14:27.1416647Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:14:27.1416769Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:14:27.1454029Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:14:27.1454409Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:14:27.1454718Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:14:27.1455014Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:14:27.1455305Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:14:27.1455470Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:14:27.1455629Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:14:27.1455791Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:14:27.1455946Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:14:27.1456010Z dist init r=3, world=4
2022-12-01T10:14:27.1456074Z dist init r=2, world=4
2022-12-01T10:14:27.1456143Z dist init r=1, world=4
2022-12-01T10:14:27.1456380Z dist init r=0, world=4
2022-12-01T10:14:27.1456485Z skip: Need at least 2 CUDA devices (1.316s)
2022-12-01T10:14:27.1456757Z   test_shard_full_optim_state_dict_nested_use_multiple_param_groups_True_wrap_alt_False_use_diff_optim_inputs_False (__main__.TestFSDPOptimState)
2022-12-01T10:14:27.1457097Z Tests :meth:`shard_full_optim_state_dict` for a non-FSDP-root model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10527
2022-12-01T10:14:27.1457240Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10528
2022-12-01T10:14:27.1457393Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10529
2022-12-01T10:14:27.1457541Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10530
2022-12-01T10:14:27.1457817Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:14:27.1458033Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:14:27.1458319Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:14:27.1458451Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:14:27.1458627Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:14:27.1458891Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:14:27.1459002Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:14:27.1459281Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:14:27.1459410Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:14:27.1459585Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:14:27.1459853Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:14:27.1459974Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:14:27.1460248Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:14:27.1460496Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:14:27.1460678Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-12-01T10:14:27.1460980Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:14:27.1461245Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:14:27.1461359Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:14:27.1461634Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:14:27.1461768Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:14:27.1461939Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:14:27.1462233Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:14:27.1462525Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:14:27.1462807Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:14:27.1462968Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:14:27.1463127Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:14:27.1463337Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:14:27.1463530Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:14:27.1463598Z dist init r=1, world=4
2022-12-01T10:14:27.1463669Z dist init r=0, world=4
2022-12-01T10:14:27.1463735Z dist init r=3, world=4
2022-12-01T10:14:27.1463794Z dist init r=2, world=4
2022-12-01T10:14:27.1463897Z skip: Need at least 2 CUDA devices (1.417s)
2022-12-01T10:14:27.1464115Z   test_shard_full_optim_state_dict_nested_use_multiple_param_groups_True_wrap_alt_False_use_diff_optim_inputs_True (__main__.TestFSDPOptimState)
2022-12-01T10:14:27.1464448Z Tests :meth:`shard_full_optim_state_dict` for a non-FSDP-root model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10583
2022-12-01T10:14:27.1464602Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10584
2022-12-01T10:14:27.1464752Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10585
2022-12-01T10:14:27.1464903Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10586
2022-12-01T10:14:27.1465176Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:14:27.1465287Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:14:27.1465569Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:14:27.1465707Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:14:27.1465884Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:14:27.1466157Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:14:27.1466281Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:14:27.1466566Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:14:27.1466705Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:14:27.1466882Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:14:27.1467138Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:14:27.1467262Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:14:27.1467539Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:14:27.1467674Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:14:27.1467850Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:14:27.1468119Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:14:27.1468245Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:14:27.1468521Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:14:27.1468654Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:14:27.1468817Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-12-01T10:14:27.1469116Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:14:27.1469415Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:14:27.1469748Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:14:27.1470045Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:14:27.1470260Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:14:27.1470426Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:14:27.1470588Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:14:27.1470750Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:14:27.1470814Z dist init r=1, world=4
2022-12-01T10:14:27.1470888Z dist init r=0, world=4
2022-12-01T10:14:27.1470961Z dist init r=3, world=4
2022-12-01T10:14:27.1471032Z dist init r=2, world=4
2022-12-01T10:14:27.1471136Z skip: Need at least 2 CUDA devices (1.417s)
2022-12-01T10:14:27.1471362Z   test_shard_full_optim_state_dict_nested_use_multiple_param_groups_True_wrap_alt_True_use_diff_optim_inputs_False (__main__.TestFSDPOptimState)
2022-12-01T10:14:27.1471702Z Tests :meth:`shard_full_optim_state_dict` for a non-FSDP-root model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10639
2022-12-01T10:14:27.1471845Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10640
2022-12-01T10:14:27.1472001Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10641
2022-12-01T10:14:27.1472154Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10642
2022-12-01T10:14:27.1472429Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:14:27.1472551Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:14:27.1472830Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:14:27.1472968Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:14:27.1473147Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:14:27.1473418Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:14:27.1473529Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:14:27.1473810Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:14:27.1473944Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:14:27.1474121Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:14:27.1474392Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:14:27.1474516Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:14:27.1474791Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:14:27.1474928Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:14:27.1475090Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-12-01T10:14:27.1475358Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:14:27.1475479Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:14:27.1475756Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:14:27.1475889Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:14:27.1476065Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:14:27.1476397Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:14:27.1476725Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:14:27.1477017Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:14:27.1477300Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:14:27.1477465Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:14:27.1477627Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:14:27.1477788Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:14:27.1477950Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:14:27.1478031Z dist init r=3, world=4
2022-12-01T10:14:27.1478107Z dist init r=1, world=4
2022-12-01T10:14:27.1478180Z dist init r=0, world=4
2022-12-01T10:14:27.1478239Z dist init r=2, world=4
2022-12-01T10:14:27.1478343Z skip: Need at least 2 CUDA devices (1.417s)
2022-12-01T10:14:27.1478565Z   test_shard_full_optim_state_dict_nested_use_multiple_param_groups_True_wrap_alt_True_use_diff_optim_inputs_True (__main__.TestFSDPOptimState)
2022-12-01T10:14:27.1478901Z Tests :meth:`shard_full_optim_state_dict` for a non-FSDP-root model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10695
2022-12-01T10:14:27.1479056Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10696
2022-12-01T10:14:27.1479211Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10697
2022-12-01T10:14:27.1479367Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10698
2022-12-01T10:14:27.1479648Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:14:27.1479775Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:14:27.1480047Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:14:27.1480182Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:14:27.1480360Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:14:27.1480632Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:14:27.1480757Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:14:27.1481036Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:14:27.1481174Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:14:27.1481354Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:14:27.1481611Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:14:27.1481734Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:14:27.1482012Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:14:27.1482147Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:14:27.1482323Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-12-01T10:14:27.1482592Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:14:27.1482751Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:14:27.1483032Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:14:27.1483198Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:14:27.1483362Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:14:27.1483661Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:14:27.1483958Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:14:27.1484249Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:14:27.1484544Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:14:27.1484710Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:14:27.1484875Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:14:27.1485040Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:14:27.1485200Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:14:27.1485264Z dist init r=0, world=4
2022-12-01T10:14:27.1485339Z dist init r=3, world=4
2022-12-01T10:14:27.1485413Z dist init r=1, world=4
2022-12-01T10:14:27.1485485Z dist init r=2, world=4
2022-12-01T10:14:27.1485589Z skip: Need at least 2 CUDA devices (1.517s)
2022-12-01T10:14:27.1485741Z   test_shard_full_optim_state_dict_transformer (__main__.TestFSDPOptimState)
2022-12-01T10:14:27.1486064Z Tests :meth:`shard_full_optim_state_dict` for an FSDP-root ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10751
2022-12-01T10:14:27.1486207Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10752
2022-12-01T10:14:27.1486366Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10753
2022-12-01T10:14:27.1486519Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10754
2022-12-01T10:14:27.1486792Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:14:27.1486916Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:14:27.1487198Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:14:27.1487333Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:14:27.1487510Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:14:27.1487783Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:14:27.1487896Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:14:27.1488177Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:14:27.1488313Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:14:27.1488488Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:14:27.1488762Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:14:27.1488886Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:14:27.1489164Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:14:27.1489329Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:14:27.1489492Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:14:27.1489792Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:14:27.1489914Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:14:27.1490192Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:14:27.1490326Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:14:27.1490503Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-12-01T10:14:27.1490804Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:14:27.1491104Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:14:27.1491397Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:14:27.1491680Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:14:27.1491844Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:14:27.1492009Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:14:27.1492170Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:14:27.1492331Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:14:27.1492407Z dist init r=3, world=4
2022-12-01T10:14:27.1492482Z dist init r=2, world=4
2022-12-01T10:14:27.1492554Z dist init r=1, world=4
2022-12-01T10:14:27.1492615Z dist init r=0, world=4
2022-12-01T10:14:27.1492720Z skip: Need at least 2 CUDA devices (1.417s)
2022-12-01T10:14:27.1492962Z   test_shard_full_optim_state_dict_unmanaged_params_state_dict_type_StateDictType_FULL_STATE_DICT_add_to_fsdp_module_False (__main__.TestFSDPOptimState)
2022-12-01T10:14:27.1493191Z Tests :meth:`shard_full_optim_state_dict` when there are unmanaged ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10807
2022-12-01T10:14:27.1493347Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10808
2022-12-01T10:14:27.1493502Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10809
2022-12-01T10:14:27.1493657Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10810
2022-12-01T10:14:27.1493930Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:14:27.1494041Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:14:27.1494322Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:14:27.1494458Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:14:27.1494635Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:14:27.1494906Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:14:27.1495028Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:14:27.1495307Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:14:27.1495441Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:14:27.1495615Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:14:27.1495902Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:14:27.1496055Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:14:27.1496333Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:14:27.1496467Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:14:27.1496641Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-12-01T10:14:27.1496937Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:14:27.1497207Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:14:27.1497329Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:14:27.1497611Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:14:27.1497733Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:14:27.1497986Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:14:27.1498281Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:14:27.1498573Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:14:27.1498869Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:14:27.1499033Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:14:27.1499197Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:14:27.1499363Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:14:27.1499525Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:14:27.1499589Z dist init r=1, world=4
2022-12-01T10:14:27.1499662Z dist init r=0, world=4
2022-12-01T10:14:27.1499734Z dist init r=2, world=4
2022-12-01T10:14:27.1499808Z dist init r=3, world=4
2022-12-01T10:14:27.1499910Z skip: Need at least 2 CUDA devices (1.417s)
2022-12-01T10:14:27.1500148Z   test_shard_full_optim_state_dict_unmanaged_params_state_dict_type_StateDictType_FULL_STATE_DICT_add_to_fsdp_module_True (__main__.TestFSDPOptimState)
2022-12-01T10:14:27.1500504Z Tests :meth:`shard_full_optim_state_dict` when there are unmanaged ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10863
2022-12-01T10:14:27.1500651Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10864
2022-12-01T10:14:27.1500809Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10865
2022-12-01T10:14:27.1500964Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10866
2022-12-01T10:14:27.1501243Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:14:27.1501366Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:14:27.1501648Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:14:27.1501785Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:14:27.1501961Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:14:27.1502234Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:14:27.1502345Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:14:27.1502684Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:14:27.1502883Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:14:27.1503060Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:14:27.1503330Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:14:27.1503455Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:14:27.1503733Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:14:27.1503871Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:14:27.1504033Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-12-01T10:14:27.1504336Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:14:27.1504607Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:14:27.1504730Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:14:27.1505012Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:14:27.1505147Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:14:27.1505321Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:14:27.1505617Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:14:27.1505917Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:14:27.1506213Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:14:27.1506366Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:14:27.1506531Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:14:27.1506695Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:14:27.1506856Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:14:27.1506932Z dist init r=0, world=4
2022-12-01T10:14:27.1507007Z dist init r=3, world=4
2022-12-01T10:14:27.1507081Z dist init r=2, world=4
2022-12-01T10:14:27.1507140Z dist init r=1, world=4
2022-12-01T10:14:27.1507245Z skip: Need at least 2 CUDA devices (1.417s)
2022-12-01T10:14:27.1507489Z   test_shard_full_optim_state_dict_unmanaged_params_state_dict_type_StateDictType_SHARDED_STATE_DICT_add_to_fsdp_module_False (__main__.TestFSDPOptimState)
2022-12-01T10:14:27.1507719Z Tests :meth:`shard_full_optim_state_dict` when there are unmanaged ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10919
2022-12-01T10:14:27.1507876Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10920
2022-12-01T10:14:27.1508029Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10921
2022-12-01T10:14:27.1508181Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10922
2022-12-01T10:14:27.1508457Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:14:27.1508582Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:14:27.1508851Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:14:27.1509020Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:14:27.1509230Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:14:27.1509503Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:14:27.1509627Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:14:27.1509908Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:14:27.1510044Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:14:27.1510221Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:14:27.1510476Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:14:27.1510603Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:14:27.1510881Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:14:27.1511018Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:14:27.1511193Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-12-01T10:14:27.1511463Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:14:27.1511587Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:14:27.1511863Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:14:27.1511998Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:14:27.1512160Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:14:27.1512464Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:14:27.1512764Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:14:27.1513058Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:14:27.1513351Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:14:27.1513517Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:14:27.1513682Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:14:27.1513844Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:14:27.1514010Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:14:27.1514075Z dist init r=3, world=4
2022-12-01T10:14:27.1514149Z dist init r=1, world=4
2022-12-01T10:14:27.1514223Z dist init r=2, world=4
2022-12-01T10:14:27.1514297Z dist init r=0, world=4
2022-12-01T10:14:27.1514401Z skip: Need at least 2 CUDA devices (1.417s)
2022-12-01T10:14:27.1514640Z   test_shard_full_optim_state_dict_unmanaged_params_state_dict_type_StateDictType_SHARDED_STATE_DICT_add_to_fsdp_module_True (__main__.TestFSDPOptimState)
2022-12-01T10:14:27.1514871Z Tests :meth:`shard_full_optim_state_dict` when there are unmanaged ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10975
2022-12-01T10:14:27.1515028Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10976
2022-12-01T10:14:27.1515170Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10977
2022-12-01T10:14:27.1515354Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10978
2022-12-01T10:14:27.1515629Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:14:27.1515798Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:14:27.1516081Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:14:27.1516216Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:14:27.1516395Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:14:27.1516668Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:14:27.1516777Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:14:27.1517057Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:14:27.1517194Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:14:27.1517371Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:14:27.1517643Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:14:27.1517764Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:14:27.1518043Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:14:27.1518178Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:14:27.1518355Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:14:27.1518611Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:14:27.1518736Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:14:27.1519017Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:14:27.1519151Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:14:27.1519324Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-12-01T10:14:27.1519622Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:14:27.1519919Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:14:27.1520214Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:14:27.1520511Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:14:27.1520662Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:14:27.1520829Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:14:27.1520990Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:14:27.1521148Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:14:27.1521225Z dist init r=1, world=4
2022-12-01T10:14:27.1521300Z dist init r=0, world=4
2022-12-01T10:14:27.1521372Z dist init r=3, world=4
2022-12-01T10:14:27.1521431Z dist init r=2, world=4
2022-12-01T10:14:27.1521536Z skip: Need at least 2 CUDA devices (1.417s)
2022-12-01T10:14:27.1521545Z 
2022-12-01T10:14:27.1521745Z ----------------------------------------------------------------------
2022-12-01T10:14:27.1521826Z Ran 50 tests in 71.191s
2022-12-01T10:14:27.1521832Z 
2022-12-01T10:14:27.1521941Z OK (skipped=50)
2022-12-01T10:14:27.1521947Z 
2022-12-01T10:14:27.1522065Z Generating XML reports...
2022-12-01T10:14:27.1522412Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_optim_state/TEST-TestFSDPOptimState-20221201101315.xml
2022-12-01T10:14:27.1522418Z 
2022-12-01T10:14:27.1522908Z ##[endgroup]
2022-12-01T10:14:27.1523275Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_optim_state (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_optim_state_8meyz2ag)
2022-12-01T10:14:27.1523280Z 
2022-12-01T10:14:27.1523479Z Running distributed/fsdp/test_fsdp_mixed_precision ... [2022-12-01 10:14:27.092051]
2022-12-01T10:14:27.1523860Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_mixed_precision.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-12-01 10:14:27.092473]
2022-12-01T10:15:37.4024965Z 
2022-12-01T10:15:37.4025525Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_mixed_precision
2022-12-01T10:15:37.4026843Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_mixed_precision (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_mixed_precision_rb0ra5d6)
2022-12-01T10:15:37.4060741Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_mixed_precision
2022-12-01T10:15:37.4061179Z 
2022-12-01T10:15:37.4061325Z Running tests...
2022-12-01T10:15:37.4061873Z ----------------------------------------------------------------------
2022-12-01T10:15:37.4062379Z   test_grads_reduced_precision (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11042
2022-12-01T10:15:37.4062802Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11043
2022-12-01T10:15:37.4063345Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:15:37.4067343Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:15:37.4068288Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:15:37.4068832Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:15:37.4069328Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:15:37.4070063Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:15:37.4070586Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:15:37.4071255Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:15:37.4101515Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:15:37.4102144Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:15:37.4103059Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:15:37.4103616Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:15:37.4104013Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:15:37.4104353Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:15:37.4104613Z dist init r=1, world=2
2022-12-01T10:15:37.4104798Z dist init r=0, world=2
2022-12-01T10:15:37.4104999Z skip: Need at least 2 CUDA devices (1.623s)
2022-12-01T10:15:37.4105607Z   test_mixed_precision_e2e_full_shard_mp_fp16_offload_false_fp32_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11070
2022-12-01T10:15:37.4106638Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11071
2022-12-01T10:15:37.4107509Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:15:37.4108073Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:15:37.4108687Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:15:37.4109041Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:15:37.4109367Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:15:37.4109843Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:15:37.4110174Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:15:37.4110611Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:15:37.4111141Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:15:37.4111693Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:15:37.4112579Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:15:37.4113601Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:15:37.4114309Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:15:37.4114836Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:15:37.4115265Z dist init r=1, world=2
2022-12-01T10:15:37.4115590Z dist init r=0, world=2
2022-12-01T10:15:37.4115945Z skip: Need at least 2 CUDA devices (1.611s)
2022-12-01T10:15:37.4116711Z   test_mixed_precision_e2e_full_shard_mp_fp16_offload_false_fp32_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11098
2022-12-01T10:15:37.4117493Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11099
2022-12-01T10:15:37.4118337Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:15:37.4118923Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:15:37.4119711Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:15:37.4120278Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:15:37.4120603Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:15:37.4121262Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:15:37.4121857Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:15:37.4122573Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:15:37.4123130Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:15:37.4123750Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:15:37.4124521Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:15:37.4125472Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:15:37.4126156Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:15:37.4126888Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:15:37.4127322Z dist init r=0, world=2
2022-12-01T10:15:37.4127650Z dist init r=1, world=2
2022-12-01T10:15:37.4127977Z skip: Need at least 2 CUDA devices (1.611s)
2022-12-01T10:15:37.4128723Z   test_mixed_precision_e2e_full_shard_mp_fp16_offload_false_fp64_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11126
2022-12-01T10:15:37.4129460Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11127
2022-12-01T10:15:37.4130279Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:15:37.4130877Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:15:37.4131671Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:15:37.4132309Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:15:37.4132773Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:15:37.4133251Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:15:37.4133582Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:15:37.4133994Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:15:37.4134338Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:15:37.4134677Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:15:37.4135172Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:15:37.4135678Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:15:37.4136067Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:15:37.4136420Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:15:37.4136669Z dist init r=1, world=2
2022-12-01T10:15:37.4136852Z dist init r=0, world=2
2022-12-01T10:15:37.4137062Z skip: Need at least 2 CUDA devices (1.611s)
2022-12-01T10:15:37.4137480Z   test_mixed_precision_e2e_full_shard_mp_fp16_offload_false_fp64_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11154
2022-12-01T10:15:37.4138026Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11155
2022-12-01T10:15:37.4138484Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:15:37.4138816Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:15:37.4139245Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:15:37.4139595Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:15:37.4139932Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:15:37.4140564Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:15:37.4140883Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:15:37.4141550Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:15:37.4142107Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:15:37.4142651Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:15:37.4143551Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:15:37.4144444Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:15:37.4145079Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:15:37.4145733Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:15:37.4145991Z dist init r=0, world=2
2022-12-01T10:15:37.4146177Z dist init r=1, world=2
2022-12-01T10:15:37.4146390Z skip: Need at least 2 CUDA devices (1.611s)
2022-12-01T10:15:37.4146823Z   test_mixed_precision_e2e_full_shard_mp_fp16_offload_true_fp32_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11182
2022-12-01T10:15:37.4147298Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11183
2022-12-01T10:15:37.4147758Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:15:37.4148092Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:15:37.4148506Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:15:37.4148852Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:15:37.4149192Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:15:37.4149645Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:15:37.4149976Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:15:37.4150406Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:15:37.4150756Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:15:37.4151081Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:15:37.4151572Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:15:37.4152376Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:15:37.4152758Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:15:37.4153093Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:15:37.4153347Z dist init r=0, world=2
2022-12-01T10:15:37.4153530Z dist init r=1, world=2
2022-12-01T10:15:37.4153728Z skip: Need at least 2 CUDA devices (1.611s)
2022-12-01T10:15:37.4154146Z   test_mixed_precision_e2e_full_shard_mp_fp16_offload_true_fp32_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11210
2022-12-01T10:15:37.4154605Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11211
2022-12-01T10:15:37.4155059Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:15:37.4155377Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:15:37.4155812Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:15:37.4156160Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:15:37.4156485Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:15:37.4156955Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:15:37.4157326Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:15:37.4157786Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:15:37.4158115Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:15:37.4158450Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:15:37.4158945Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:15:37.4159466Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:15:37.4159839Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:15:37.4160188Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:15:37.4160458Z dist init r=1, world=2
2022-12-01T10:15:37.4160633Z dist init r=0, world=2
2022-12-01T10:15:37.4160845Z skip: Need at least 2 CUDA devices (1.611s)
2022-12-01T10:15:37.4161288Z   test_mixed_precision_e2e_full_shard_mp_fp16_offload_true_fp64_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11238
2022-12-01T10:15:37.4161759Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11239
2022-12-01T10:15:37.4162201Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:15:37.4162531Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:15:37.4162962Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:15:37.4163293Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:15:37.4163636Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:15:37.4164108Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:15:37.4164434Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:15:37.4164848Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:15:37.4165193Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:15:37.4165532Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:15:37.4166030Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:15:37.4166538Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:15:37.4166923Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:15:37.4167272Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:15:37.4167520Z dist init r=0, world=2
2022-12-01T10:15:37.4167705Z dist init r=1, world=2
2022-12-01T10:15:37.4167916Z skip: Need at least 2 CUDA devices (1.611s)
2022-12-01T10:15:37.4168330Z   test_mixed_precision_e2e_full_shard_mp_fp16_offload_true_fp64_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11266
2022-12-01T10:15:37.4168765Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11267
2022-12-01T10:15:37.4169220Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:15:37.4169548Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:15:37.4170002Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:15:37.4170381Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:15:37.4170714Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:15:37.4171185Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:15:37.4171499Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:15:37.4171928Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:15:37.4172276Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:15:37.4172611Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:15:37.4173098Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:15:37.4173626Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:15:37.4174014Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:15:37.4174349Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:15:37.4174609Z dist init r=1, world=2
2022-12-01T10:15:37.4174797Z dist init r=0, world=2
2022-12-01T10:15:37.4175008Z skip: Need at least 2 CUDA devices (1.611s)
2022-12-01T10:15:37.4175434Z   test_mixed_precision_e2e_full_shard_mp_no_mp_offload_false_fp32_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11294
2022-12-01T10:15:37.4175913Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11295
2022-12-01T10:15:37.4176370Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:15:37.4176693Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:15:37.4177121Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:15:37.4177472Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:15:37.4177896Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:15:37.4178357Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:15:37.4178685Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:15:37.4179113Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:15:37.4179457Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:15:37.4179779Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:15:37.4180272Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:15:37.4180959Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:15:37.4181333Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:15:37.4181685Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:15:37.4181943Z dist init r=0, world=2
2022-12-01T10:15:37.4182127Z dist init r=1, world=2
2022-12-01T10:15:37.4182324Z skip: Need at least 2 CUDA devices (1.611s)
2022-12-01T10:15:37.4182823Z   test_mixed_precision_e2e_full_shard_mp_no_mp_offload_false_fp32_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11322
2022-12-01T10:15:37.4183336Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11323
2022-12-01T10:15:37.4183780Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:15:37.4184116Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:15:37.4184545Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:15:37.4184891Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:15:37.4185215Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:15:37.4185681Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:15:37.4186015Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:15:37.4186444Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:15:37.4186777Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:15:37.4187110Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:15:37.4187598Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:15:37.4188104Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:15:37.4188495Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:15:37.4188843Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:15:37.4189105Z dist init r=0, world=2
2022-12-01T10:15:37.4189273Z dist init r=1, world=2
2022-12-01T10:15:37.4189484Z skip: Need at least 2 CUDA devices (1.611s)
2022-12-01T10:15:37.4189926Z   test_mixed_precision_e2e_full_shard_mp_no_mp_offload_false_fp64_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11350
2022-12-01T10:15:37.4190385Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11351
2022-12-01T10:15:37.4190840Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:15:37.4191171Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:15:37.4191600Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:15:37.4191933Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:15:37.4192273Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:15:37.4192748Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:15:37.4193077Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:15:37.4193490Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:15:37.4193835Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:15:37.4194174Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:15:37.4194654Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:15:37.4195174Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:15:37.4195596Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:15:37.4195973Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:15:37.4196221Z dist init r=0, world=2
2022-12-01T10:15:37.4196406Z dist init r=1, world=2
2022-12-01T10:15:37.4196618Z skip: Need at least 2 CUDA devices (1.611s)
2022-12-01T10:15:37.4197021Z   test_mixed_precision_e2e_full_shard_mp_no_mp_offload_false_fp64_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11378
2022-12-01T10:15:37.4197476Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11379
2022-12-01T10:15:37.4197932Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:15:37.4198263Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:15:37.4198679Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:15:37.4199025Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:15:37.4199362Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:15:37.4199830Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:15:37.4200145Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:15:37.4200573Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:15:37.4200918Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:15:37.4201237Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:15:37.4201736Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:15:37.4202261Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:15:37.4202650Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:15:37.4202988Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:15:37.4203243Z dist init r=1, world=2
2022-12-01T10:15:37.4203426Z dist init r=0, world=2
2022-12-01T10:15:37.4203622Z skip: Need at least 2 CUDA devices (1.611s)
2022-12-01T10:15:37.4204062Z   test_mixed_precision_e2e_full_shard_mp_no_mp_offload_true_fp32_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11406
2022-12-01T10:15:37.4204537Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11407
2022-12-01T10:15:37.4204995Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:15:37.4205316Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:15:37.4205747Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:15:37.4206093Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:15:37.4206439Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:15:37.4206896Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:15:37.4207225Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:15:37.4207649Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:15:37.4208017Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:15:37.4208426Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:15:37.4208931Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:15:37.4209452Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:15:37.4209829Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:15:37.4210178Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:15:37.4210441Z dist init r=0, world=2
2022-12-01T10:15:37.4210612Z dist init r=1, world=2
2022-12-01T10:15:37.4210822Z skip: Need at least 2 CUDA devices (1.611s)
2022-12-01T10:15:37.4211241Z   test_mixed_precision_e2e_full_shard_mp_no_mp_offload_true_fp32_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11434
2022-12-01T10:15:37.4211693Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11435
2022-12-01T10:15:37.4212135Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:15:37.4212465Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:15:37.4212891Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:15:37.4213236Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:15:37.4213559Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:15:37.4214034Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:15:37.4214364Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:15:37.4214783Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:15:37.4215129Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:15:37.4215467Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:15:37.4215960Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:15:37.4216463Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:15:37.4216848Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:15:37.4217196Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:15:37.4217446Z dist init r=1, world=2
2022-12-01T10:15:37.4217629Z dist init r=0, world=2
2022-12-01T10:15:37.4217933Z skip: Need at least 2 CUDA devices (1.611s)
2022-12-01T10:15:37.4218376Z   test_mixed_precision_e2e_full_shard_mp_no_mp_offload_true_fp64_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11462
2022-12-01T10:15:37.4218832Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11463
2022-12-01T10:15:37.4219294Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:15:37.4219627Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:15:37.4220055Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:15:37.4220526Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:15:37.4220935Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:15:37.4221454Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:15:37.4221772Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:15:37.4222198Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:15:37.4222546Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:15:37.4222884Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:15:37.4223364Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:15:37.4223885Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:15:37.4224278Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:15:37.4224632Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:15:37.4224880Z dist init r=0, world=2
2022-12-01T10:15:37.4225065Z dist init r=1, world=2
2022-12-01T10:15:37.4225277Z skip: Need at least 2 CUDA devices (1.612s)
2022-12-01T10:15:37.4225681Z   test_mixed_precision_e2e_full_shard_mp_no_mp_offload_true_fp64_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11490
2022-12-01T10:15:37.4226132Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11491
2022-12-01T10:15:37.4226594Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:15:37.4226928Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:15:37.4227344Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:15:37.4227695Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:15:37.4228040Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:15:37.4228500Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:15:37.4228830Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:15:37.4229257Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:15:37.4229600Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:15:37.4229925Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:15:37.4230429Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:15:37.4230951Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:15:37.4231327Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:15:37.4231678Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:15:37.4231935Z dist init r=0, world=2
2022-12-01T10:15:37.4232118Z dist init r=1, world=2
2022-12-01T10:15:37.4232318Z skip: Need at least 2 CUDA devices (1.611s)
2022-12-01T10:15:37.4232774Z   test_mixed_precision_e2e_full_shard_mp_only_param_and_buf_offload_false_fp32_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11518
2022-12-01T10:15:37.4233342Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11519
2022-12-01T10:15:37.4233829Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:15:37.4234177Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:15:37.4234611Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:15:37.4234959Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:15:37.4235297Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:15:37.4235754Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:15:37.4236083Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:15:37.4236510Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:15:37.4236847Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:15:37.4237192Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:15:37.4237689Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:15:37.4238211Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:15:37.4238585Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:15:37.4238934Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:15:37.4239195Z dist init r=1, world=2
2022-12-01T10:15:37.4239377Z dist init r=0, world=2
2022-12-01T10:15:37.4239575Z skip: Need at least 2 CUDA devices (1.611s)
2022-12-01T10:15:37.4240008Z   test_mixed_precision_e2e_full_shard_mp_only_param_and_buf_offload_false_fp32_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11546
2022-12-01T10:15:37.4240479Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11547
2022-12-01T10:15:37.4240919Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:15:37.4241250Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:15:37.4241683Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:15:37.4242032Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:15:37.4242356Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:15:37.4242825Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:15:37.4243156Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:15:37.4243575Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:15:37.4243922Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:15:37.4244264Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:15:37.4244768Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:15:37.4245276Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:15:37.4245671Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:15:37.4246023Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:15:37.4246318Z dist init r=1, world=2
2022-12-01T10:15:37.4246489Z dist init r=0, world=2
2022-12-01T10:15:37.4246733Z skip: Need at least 2 CUDA devices (1.611s)
2022-12-01T10:15:37.4247189Z   test_mixed_precision_e2e_full_shard_mp_only_param_and_buf_offload_false_fp64_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11574
2022-12-01T10:15:37.4247668Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11575
2022-12-01T10:15:37.4248123Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:15:37.4248452Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:15:37.4248885Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:15:37.4249221Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:15:37.4249565Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:15:37.4250035Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:15:37.4250350Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:15:37.4250777Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:15:37.4251123Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:15:37.4251459Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:15:37.4251942Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:15:37.4252465Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:15:37.4252858Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:15:37.4253211Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:15:37.4253457Z dist init r=1, world=2
2022-12-01T10:15:37.4253639Z dist init r=0, world=2
2022-12-01T10:15:37.4253849Z skip: Need at least 2 CUDA devices (1.511s)
2022-12-01T10:15:37.4254266Z   test_mixed_precision_e2e_full_shard_mp_only_param_and_buf_offload_false_fp64_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11602
2022-12-01T10:15:37.4254730Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11603
2022-12-01T10:15:37.4255192Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:15:37.4255522Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:15:37.4255942Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:15:37.4256297Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:15:37.4256633Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:15:37.4257091Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:15:37.4257423Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:15:37.4257976Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:15:37.4258323Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:15:37.4258642Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:15:37.4259187Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:15:37.4259744Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:15:37.4260132Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:15:37.4260672Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:15:37.4260931Z dist init r=0, world=2
2022-12-01T10:15:37.4261115Z dist init r=1, world=2
2022-12-01T10:15:37.4261312Z skip: Need at least 2 CUDA devices (1.611s)
2022-12-01T10:15:37.4261770Z   test_mixed_precision_e2e_full_shard_mp_only_param_and_buf_offload_true_fp32_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11630
2022-12-01T10:15:37.4262264Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11631
2022-12-01T10:15:37.4262724Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:15:37.4263043Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:15:37.4263472Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:15:37.4263822Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:15:37.4264149Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:15:37.4264623Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:15:37.4264950Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:15:37.4265378Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:15:37.4265714Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:15:37.4266052Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:15:37.4266547Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:15:37.4267067Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:15:37.4267444Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:15:37.4267793Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:15:37.4268055Z dist init r=0, world=2
2022-12-01T10:15:37.4268224Z dist init r=1, world=2
2022-12-01T10:15:37.4268434Z skip: Need at least 2 CUDA devices (1.611s)
2022-12-01T10:15:37.4268868Z   test_mixed_precision_e2e_full_shard_mp_only_param_and_buf_offload_true_fp32_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11658
2022-12-01T10:15:37.4269328Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11659
2022-12-01T10:15:37.4269772Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:15:37.4270104Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:15:37.4270536Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:15:37.4270887Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:15:37.4271211Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:15:37.4271747Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:15:37.4272077Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:15:37.4272560Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:15:37.4272906Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:15:37.4273245Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:15:37.4273736Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:15:37.4274243Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:15:37.4274631Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:15:37.4274980Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:15:37.4275229Z dist init r=1, world=2
2022-12-01T10:15:37.4275418Z dist init r=0, world=2
2022-12-01T10:15:37.4275631Z skip: Need at least 2 CUDA devices (1.611s)
2022-12-01T10:15:37.4276085Z   test_mixed_precision_e2e_full_shard_mp_only_param_and_buf_offload_true_fp64_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11686
2022-12-01T10:15:37.4276551Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11687
2022-12-01T10:15:37.4277005Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:15:37.4277336Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:15:37.4277768Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:15:37.4278102Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:15:37.4278439Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:15:37.4278911Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:15:37.4279225Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:15:37.4279654Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:15:37.4280002Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:15:37.4280337Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:15:37.4280823Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:15:37.4281345Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:15:37.4281733Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:15:37.4282085Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:15:37.4282330Z dist init r=0, world=2
2022-12-01T10:15:37.4282515Z dist init r=1, world=2
2022-12-01T10:15:37.4282725Z skip: Need at least 2 CUDA devices (1.611s)
2022-12-01T10:15:37.4283138Z   test_mixed_precision_e2e_full_shard_mp_only_param_and_buf_offload_true_fp64_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11714
2022-12-01T10:15:37.4283598Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11715
2022-12-01T10:15:37.4284053Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:15:37.4284417Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:15:37.4284834Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:15:37.4285212Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:15:37.4285552Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:15:37.4286005Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:15:37.4286332Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:15:37.4286758Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:15:37.4287103Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:15:37.4287425Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:15:37.4287925Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:15:37.4288445Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:15:37.4288835Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:15:37.4289177Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:15:37.4289435Z dist init r=1, world=2
2022-12-01T10:15:37.4289621Z dist init r=0, world=2
2022-12-01T10:15:37.4289819Z skip: Need at least 2 CUDA devices (1.611s)
2022-12-01T10:15:37.4290269Z   test_mixed_precision_e2e_full_shard_mp_only_reduce_offload_false_fp32_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11742
2022-12-01T10:15:37.4290751Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11743
2022-12-01T10:15:37.4291206Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:15:37.4291525Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:15:37.4291953Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:15:37.4292300Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:15:37.4292625Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:15:37.4293099Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:15:37.4293428Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:15:37.4293858Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:15:37.4294190Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:15:37.4294530Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:15:37.4295030Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:15:37.4295545Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:15:37.4295917Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:15:37.4296269Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:15:37.4296531Z dist init r=0, world=2
2022-12-01T10:15:37.4296701Z dist init r=1, world=2
2022-12-01T10:15:37.4296911Z skip: Need at least 2 CUDA devices (1.611s)
2022-12-01T10:15:37.4297373Z   test_mixed_precision_e2e_full_shard_mp_only_reduce_offload_false_fp32_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11770
2022-12-01T10:15:37.4297940Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11771
2022-12-01T10:15:37.4298389Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:15:37.4298724Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:15:37.4299153Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:15:37.4299489Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:15:37.4299827Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:15:37.4300298Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:15:37.4300931Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:15:37.4301356Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:15:37.4301709Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:15:37.4302045Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:15:37.4302524Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:15:37.4303044Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:15:37.4303435Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:15:37.4303785Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:15:37.4304036Z dist init r=1, world=2
2022-12-01T10:15:37.4304226Z dist init r=0, world=2
2022-12-01T10:15:37.4304434Z skip: Need at least 2 CUDA devices (1.611s)
2022-12-01T10:15:37.4304865Z   test_mixed_precision_e2e_full_shard_mp_only_reduce_offload_false_fp64_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11798
2022-12-01T10:15:37.4305347Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11799
2022-12-01T10:15:37.4305799Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:15:37.4306129Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:15:37.4306545Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:15:37.4306895Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:15:37.4307233Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:15:37.4307701Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:15:37.4308016Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:15:37.4308441Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:15:37.4308786Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:15:37.4309109Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:15:37.4309604Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:15:37.4310204Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:15:37.4310640Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:15:37.4310978Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:15:37.4311234Z dist init r=0, world=2
2022-12-01T10:15:37.4311419Z dist init r=1, world=2
2022-12-01T10:15:37.4311618Z skip: Need at least 2 CUDA devices (1.611s)
2022-12-01T10:15:37.4312039Z   test_mixed_precision_e2e_full_shard_mp_only_reduce_offload_false_fp64_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11826
2022-12-01T10:15:37.4312490Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11827
2022-12-01T10:15:37.4312945Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:15:37.4313262Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:15:37.4313693Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:15:37.4314041Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:15:37.4314379Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:15:37.4314835Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:15:37.4315162Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:15:37.4315589Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:15:37.4315918Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:15:37.4316254Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:15:37.4316749Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:15:37.4317271Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:15:37.4317645Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:15:37.4317999Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:15:37.4318258Z dist init r=1, world=2
2022-12-01T10:15:37.4318426Z dist init r=0, world=2
2022-12-01T10:15:37.4318638Z skip: Need at least 2 CUDA devices (1.611s)
2022-12-01T10:15:37.4319081Z   test_mixed_precision_e2e_full_shard_mp_only_reduce_offload_true_fp32_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11854
2022-12-01T10:15:37.4319552Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11855
2022-12-01T10:15:37.4319997Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:15:37.4320327Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:15:37.4320758Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:15:37.4321107Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:15:37.4321432Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:15:37.4321900Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:15:37.4322226Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:15:37.4322675Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:15:37.4323022Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:15:37.4323387Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:15:37.4323883Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:15:37.4324388Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:15:37.4324774Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:15:37.4325126Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:15:37.4325386Z dist init r=1, world=2
2022-12-01T10:15:37.4325554Z dist init r=0, world=2
2022-12-01T10:15:37.4325762Z skip: Need at least 2 CUDA devices (1.511s)
2022-12-01T10:15:37.4326194Z   test_mixed_precision_e2e_full_shard_mp_only_reduce_offload_true_fp32_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11882
2022-12-01T10:15:37.4326634Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11883
2022-12-01T10:15:37.4327086Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:15:37.4327417Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:15:37.4327849Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:15:37.4328184Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:15:37.4328523Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:15:37.4328997Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:15:37.4329311Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:15:37.4329743Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:15:37.4330088Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:15:37.4330424Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:15:37.4330903Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:15:37.4331427Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:15:37.4331818Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:15:37.4332167Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:15:37.4332414Z dist init r=1, world=2
2022-12-01T10:15:37.4332600Z dist init r=0, world=2
2022-12-01T10:15:37.4332811Z skip: Need at least 2 CUDA devices (1.611s)
2022-12-01T10:15:37.4333247Z   test_mixed_precision_e2e_full_shard_mp_only_reduce_offload_true_fp64_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11910
2022-12-01T10:15:37.4333726Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11911
2022-12-01T10:15:37.4334182Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:15:37.4334514Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:15:37.4334928Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:15:37.4335307Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:15:37.4335647Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:15:37.4336134Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:15:37.4336461Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:15:37.4336887Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:15:37.4337229Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:15:37.4337550Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:15:37.4338141Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:15:37.4338682Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:15:37.4339074Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:15:37.4339409Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:15:37.4339665Z dist init r=0, world=2
2022-12-01T10:15:37.4339848Z dist init r=1, world=2
2022-12-01T10:15:37.4340046Z skip: Need at least 2 CUDA devices (1.612s)
2022-12-01T10:15:37.4340613Z   test_mixed_precision_e2e_full_shard_mp_only_reduce_offload_true_fp64_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11938
2022-12-01T10:15:37.4341071Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11939
2022-12-01T10:15:37.4341531Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:15:37.4341850Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:15:37.4342282Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:15:37.4342635Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:15:37.4342961Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:15:37.4343435Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:15:37.4343768Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:15:37.4344195Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:15:37.4344525Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:15:37.4344864Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:15:37.4345365Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:15:37.4345886Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:15:37.4346265Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:15:37.4346613Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:15:37.4346875Z dist init r=1, world=2
2022-12-01T10:15:37.4347044Z dist init r=0, world=2
2022-12-01T10:15:37.4347254Z skip: Need at least 2 CUDA devices (1.611s)
2022-12-01T10:15:37.4347656Z   test_mixed_precision_no_reshard_after_forward (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11966
2022-12-01T10:15:37.4348090Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11967
2022-12-01T10:15:37.4348600Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:15:37.4348966Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:15:37.4349399Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:15:37.4349733Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:15:37.4350071Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:15:37.4350541Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:15:37.4350868Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:15:37.4351280Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:15:37.4351630Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:15:37.4351973Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:15:37.4352465Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:15:37.4352971Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:15:37.4353360Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:15:37.4353711Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:15:37.4353961Z dist init r=1, world=2
2022-12-01T10:15:37.4354143Z dist init r=0, world=2
2022-12-01T10:15:37.4354353Z skip: Need at least 2 CUDA devices (1.611s)
2022-12-01T10:15:37.4354629Z   test_mixed_precision_resnet (__main__.TestFSDPMixedPrecisionSharded)
2022-12-01T10:15:37.4354969Z End to end test to ensure mixed precision + auto_wrap works ... skip: no torchvision (0.001s)
2022-12-01T10:15:37.4355412Z   test_mp_batchnorm_convert_sync_bn_False (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11994
2022-12-01T10:15:37.4355843Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11995
2022-12-01T10:15:37.4356283Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:15:37.4356614Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:15:37.4357044Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:15:37.4357391Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:15:37.4357713Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:15:37.4358180Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:15:37.4358511Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:15:37.4358925Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:15:37.4359269Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:15:37.4359609Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:15:37.4360101Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:15:37.4360605Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:15:37.4361027Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:15:37.4361422Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:15:37.4361682Z dist init r=1, world=2
2022-12-01T10:15:37.4361853Z dist init r=0, world=2
2022-12-01T10:15:37.4362065Z skip: Need at least 2 CUDA devices (1.612s)
2022-12-01T10:15:37.4362454Z   test_mp_batchnorm_convert_sync_bn_True (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12022
2022-12-01T10:15:37.4362866Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12023
2022-12-01T10:15:37.4363321Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:15:37.4363654Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:15:37.4364085Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:15:37.4364418Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:15:37.4364758Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:15:37.4365226Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:15:37.4365539Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:15:37.4365966Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:15:37.4366309Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:15:37.4366644Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:15:37.4367125Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:15:37.4367642Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:15:37.4368029Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:15:37.4368375Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:15:37.4368625Z dist init r=0, world=2
2022-12-01T10:15:37.4368808Z dist init r=1, world=2
2022-12-01T10:15:37.4369018Z skip: Need at least 2 CUDA devices (1.612s)
2022-12-01T10:15:37.4369387Z   test_mp_embedding_default (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12050
2022-12-01T10:15:37.4369800Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12051
2022-12-01T10:15:37.4370253Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:15:37.4370571Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:15:37.4371002Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:15:37.4371349Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:15:37.4371685Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:15:37.4372138Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:15:37.4372470Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:15:37.4372900Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:15:37.4373246Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:15:37.4373606Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:15:37.4374106Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:15:37.4374656Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:15:37.4375031Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:15:37.4375380Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:15:37.4375636Z dist init r=1, world=2
2022-12-01T10:15:37.4375818Z dist init r=0, world=2
2022-12-01T10:15:37.4376014Z skip: Need at least 2 CUDA devices (1.610s)
2022-12-01T10:15:37.4376405Z   test_mp_embedding_only_params_and_bufs (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12078
2022-12-01T10:15:37.4376834Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12079
2022-12-01T10:15:37.4377273Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:15:37.4377607Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:15:37.4378116Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:15:37.4378462Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:15:37.4378784Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:15:37.4379253Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:15:37.4379584Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:15:37.4380012Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:15:37.4380340Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:15:37.4380817Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:15:37.4381318Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:15:37.4381824Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:15:37.4382212Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:15:37.4382561Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:15:37.4382819Z dist init r=1, world=2
2022-12-01T10:15:37.4382989Z dist init r=0, world=2
2022-12-01T10:15:37.4383199Z skip: Need at least 2 CUDA devices (1.611s)
2022-12-01T10:15:37.4383594Z   test_mp_embedding_params_and_reduce_diff (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12106
2022-12-01T10:15:37.4384009Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12107
2022-12-01T10:15:37.4384461Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:15:37.4384794Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:15:37.4385222Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:15:37.4385555Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:15:37.4385890Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:15:37.4386360Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:15:37.4386732Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:15:37.4387198Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:15:37.4387539Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:15:37.4387871Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:15:37.4388347Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:15:37.4388864Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:15:37.4389250Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:15:37.4389596Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:15:37.4389846Z dist init r=1, world=2
2022-12-01T10:15:37.4390030Z dist init r=0, world=2
2022-12-01T10:15:37.4390242Z skip: Need at least 2 CUDA devices (1.611s)
2022-12-01T10:15:37.4390611Z   test_mp_embedding_reduce (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12134
2022-12-01T10:15:37.4391030Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12135
2022-12-01T10:15:37.4391482Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:15:37.4391812Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:15:37.4392225Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:15:37.4392571Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:15:37.4392908Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:15:37.4393364Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:15:37.4393694Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:15:37.4394124Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:15:37.4394467Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:15:37.4394789Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:15:37.4395283Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:15:37.4395805Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:15:37.4396194Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:15:37.4396530Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:15:37.4396785Z dist init r=1, world=2
2022-12-01T10:15:37.4396967Z dist init r=0, world=2
2022-12-01T10:15:37.4397164Z skip: Need at least 2 CUDA devices (1.611s)
2022-12-01T10:15:37.4397546Z   test_grads_reduced_precision (__main__.TestFSDPMixedPrecisionUnsharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12162
2022-12-01T10:15:37.4398086Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:15:37.4398414Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:15:37.4398828Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:15:37.4399177Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:15:37.4399554Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:15:37.4400065Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-12-01T10:15:37.4400454Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:15:37.4400716Z dist init r=0, world=1
2022-12-01T10:15:37.4400928Z skip: Need at least 1 CUDA device (1.507s)
2022-12-01T10:15:37.4401303Z   test_mixed_precision_e2e_full_shard (__main__.TestFSDPMixedPrecisionUnsharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12176
2022-12-01T10:15:37.4401845Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:15:37.4402174Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:15:37.4402588Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:15:37.4402933Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:15:37.4403269Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:15:37.4403762Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-12-01T10:15:37.4404138Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:15:37.4404399Z dist init r=0, world=1
2022-12-01T10:15:37.4404607Z skip: Need at least 1 CUDA device (1.507s)
2022-12-01T10:15:37.4404996Z   test_mixed_precision_no_reshard_after_forward (__main__.TestFSDPMixedPrecisionUnsharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12190
2022-12-01T10:15:37.4405555Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:15:37.4405886Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:15:37.4406316Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:15:37.4406648Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:15:37.4406982Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:15:37.4407479Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-12-01T10:15:37.4407868Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:15:37.4408116Z dist init r=0, world=1
2022-12-01T10:15:37.4408323Z skip: Need at least 1 CUDA device (1.507s)
2022-12-01T10:15:37.4408461Z 
2022-12-01T10:15:37.4408666Z ----------------------------------------------------------------------
2022-12-01T10:15:37.4408900Z Ran 44 tests in 68.783s
2022-12-01T10:15:37.4409018Z 
2022-12-01T10:15:37.4409093Z OK (skipped=44)
2022-12-01T10:15:37.4409204Z 
2022-12-01T10:15:37.4409290Z Generating XML reports...
2022-12-01T10:15:37.4409801Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_mixed_precision/TEST-TestFSDPMixedPrecisionSharded-20221201101428.xml
2022-12-01T10:15:37.4410474Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_mixed_precision/TEST-TestFSDPMixedPrecisionUnsharded-20221201101428.xml
2022-12-01T10:15:37.4410778Z 
2022-12-01T10:15:37.4411182Z ##[endgroup]
2022-12-01T10:15:37.4411670Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_mixed_precision (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_mixed_precision_rb0ra5d6)
2022-12-01T10:15:37.4411952Z 
2022-12-01T10:15:37.4412132Z Running distributed/fsdp/test_wrap ... [2022-12-01 10:15:37.403636]
2022-12-01T10:15:37.4412702Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_wrap.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-12-01 10:15:37.403957]
2022-12-01T10:16:11.6828669Z 
2022-12-01T10:16:11.6829525Z Expand the folded group to see the log file of distributed/fsdp/test_wrap
2022-12-01T10:16:11.6830609Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_wrap (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_wrap_xsqgz3g7)
2022-12-01T10:16:11.6831433Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_wrap
2022-12-01T10:16:11.6831722Z 
2022-12-01T10:16:11.6831831Z Running tests...
2022-12-01T10:16:11.6832287Z ----------------------------------------------------------------------
2022-12-01T10:16:11.6833008Z   test_always_wrap (__main__.TestAutoWrap)
2022-12-01T10:16:11.6833473Z Test to ensure that if `always_wrap_policy` is ... skip: Test Requires CUDA (0.001s)
2022-12-01T10:16:11.6834166Z   test_always_wrap_with_ignored_modules_wrap_method_WrapMethod_FSDP_CTOR (__main__.TestAutoWrap) ... skip: Requires at least 2 GPUs (0.001s)
2022-12-01T10:16:11.6834879Z   test_always_wrap_with_ignored_modules_wrap_method_WrapMethod_WRAP_API (__main__.TestAutoWrap) ... skip: Requires at least 2 GPUs (0.001s)
2022-12-01T10:16:11.6835418Z   test_auto_wrap_api (__main__.TestAutoWrap)
2022-12-01T10:16:11.6835948Z Test to ensure with auto wrap, we wrap child modules correctly based on the min_num_params. ... skip: Requires at least 2 GPUs (0.000s)
2022-12-01T10:16:11.6836484Z   test_auto_wrap_preset_exclude_wrap (__main__.TestAutoWrap)
2022-12-01T10:16:11.6837082Z Test to ensure excluded modules are not wrapped, regardless if the total param size is greater than the ... skip: Requires at least 2 GPUs (0.001s)
2022-12-01T10:16:11.6837721Z   test_auto_wrap_preset_exclude_wrap_include_children (__main__.TestAutoWrap)
2022-12-01T10:16:11.6838337Z Test to ensure excluded modules are not wrapped, but children are if param size is greater than ... skip: Requires at least 2 GPUs (0.000s)
2022-12-01T10:16:11.6838883Z   test_auto_wrap_preset_force_leaf (__main__.TestAutoWrap)
2022-12-01T10:16:11.6839731Z Test to ensure force-leaf modules are not wrapped, and children are not wrapped. The ... skip: Requires at least 2 GPUs (0.001s)
2022-12-01T10:16:11.6840224Z   test_auto_wrap_preset_force_leaf_custom (__main__.TestAutoWrap)
2022-12-01T10:16:11.6840788Z Test to ensure force-leaf modules are not wrapped. ... skip: Requires at least 2 GPUs (0.001s)
2022-12-01T10:16:11.6841439Z   test_auto_wrap_smoke_test_cuda_init_mode_CUDAInitMode_CUDA_AFTER_cpu_offload_CPUOffload(offload_params=False)_use_device_id_False (__main__.TestAutoWrap) ... skip: Test Requires CUDA (0.002s)
2022-12-01T10:16:11.6842202Z   test_auto_wrap_smoke_test_cuda_init_mode_CUDAInitMode_CUDA_AFTER_cpu_offload_CPUOffload(offload_params=False)_use_device_id_True (__main__.TestAutoWrap) ... skip: Test Requires CUDA (0.001s)
2022-12-01T10:16:11.6843053Z   test_auto_wrap_smoke_test_cuda_init_mode_CUDAInitMode_CUDA_AFTER_cpu_offload_CPUOffload(offload_params=True)_use_device_id_False (__main__.TestAutoWrap) ... skip: Test Requires CUDA (0.001s)
2022-12-01T10:16:11.6888137Z   test_auto_wrap_smoke_test_cuda_init_mode_CUDAInitMode_CUDA_AFTER_cpu_offload_CPUOffload(offload_params=True)_use_device_id_True (__main__.TestAutoWrap) ... skip: Test Requires CUDA (0.001s)
2022-12-01T10:16:11.6889136Z   test_auto_wrap_smoke_test_cuda_init_mode_CUDAInitMode_CUDA_BEFORE_cpu_offload_CPUOffload(offload_params=False)_use_device_id_False (__main__.TestAutoWrap) ... skip: Test Requires CUDA (0.001s)
2022-12-01T10:16:11.6889950Z   test_auto_wrap_smoke_test_cuda_init_mode_CUDAInitMode_CUDA_BEFORE_cpu_offload_CPUOffload(offload_params=False)_use_device_id_True (__main__.TestAutoWrap) ... skip: Test Requires CUDA (0.002s)
2022-12-01T10:16:11.6890764Z   test_auto_wrap_smoke_test_cuda_init_mode_CUDAInitMode_CUDA_BEFORE_cpu_offload_CPUOffload(offload_params=True)_use_device_id_False (__main__.TestAutoWrap) ... skip: Test Requires CUDA (0.001s)
2022-12-01T10:16:11.6891845Z   test_auto_wrap_smoke_test_cuda_init_mode_CUDAInitMode_CUDA_BEFORE_cpu_offload_CPUOffload(offload_params=True)_use_device_id_True (__main__.TestAutoWrap) ... skip: Test Requires CUDA (0.001s)
2022-12-01T10:16:11.6892626Z   test_auto_wrap_with_ignored_modules_wrap_method_WrapMethod_FSDP_CTOR (__main__.TestAutoWrap) ... skip: Requires at least 2 GPUs (0.001s)
2022-12-01T10:16:11.6893291Z   test_auto_wrap_with_ignored_modules_wrap_method_WrapMethod_WRAP_API (__main__.TestAutoWrap) ... skip: Requires at least 2 GPUs (0.001s)
2022-12-01T10:16:11.6893883Z   test_transformer_auto_wrap_policy (__main__.TestAutoWrap)
2022-12-01T10:16:11.6894338Z Tests the ``transformer_auto_wrap_policy``. ... skip: Requires at least 2 GPUs (0.001s)
2022-12-01T10:16:11.6894868Z   test_wrap_disabled_outside_context (__main__.TestAutoWrap) ... skip: Requires at least 2 GPUs (0.001s)
2022-12-01T10:16:11.6895388Z   test_wrap_override_defaults (__main__.TestAutoWrap) ... skip: Requires at least 2 GPUs (0.001s)
2022-12-01T10:16:11.6895896Z   test_wrap_wrap_method_WrapMethod_FSDP_CTOR (__main__.TestAutoWrap) ... skip: Requires at least 2 GPUs (0.001s)
2022-12-01T10:16:11.6896291Z   test_wrap_wrap_method_WrapMethod_WRAP_API (__main__.TestAutoWrap) ... skip: Requires at least 2 GPUs (0.001s)
2022-12-01T10:16:11.6896615Z   test_bn_always_wrapped_individually (__main__.TestFSDPWrap)
2022-12-01T10:16:11.6897000Z Ensures that by using _or_policy with _wrap_batchnorm_individually, even ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12215
2022-12-01T10:16:11.6897391Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12216
2022-12-01T10:16:11.6897804Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12217
2022-12-01T10:16:11.6898132Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 12218
2022-12-01T10:16:11.6898671Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:16:11.6899009Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:16:11.6899453Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:16:11.6899802Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:16:11.6900129Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:16:11.6900782Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:16:11.6901118Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:16:11.6901547Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:16:11.6901878Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:16:11.6902215Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:16:11.6902684Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:16:11.6902996Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:16:11.6903421Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:16:11.6903768Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:16:11.6904099Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:16:11.6904551Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:16:11.6904875Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:16:11.6905417Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:16:11.6905765Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:16:11.6906136Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-12-01T10:16:11.6906635Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:16:11.6907157Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:16:11.6907663Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:16:11.6908170Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:16:11.6908553Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:16:11.6908902Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:16:11.6909234Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:16:11.6909570Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:16:11.6909825Z dist init r=2, world=4
2022-12-01T10:16:11.6909992Z dist init r=1, world=4
2022-12-01T10:16:11.6910168Z dist init r=0, world=4
2022-12-01T10:16:11.6910341Z dist init r=3, world=4
2022-12-01T10:16:11.6910531Z skip: Need at least 2 CUDA devices (1.757s)
2022-12-01T10:16:11.6910844Z   test_error_already_wrapped_nested_False_cuda_init_mode_CUDAInitMode_CUDA_AFTER (__main__.TestFSDPWrap)
2022-12-01T10:16:11.6911270Z Test that an error is raised if we attempt to wrap when submodules are ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12271
2022-12-01T10:16:11.6911667Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12272
2022-12-01T10:16:11.6911984Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12273
2022-12-01T10:16:11.6912311Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 12274
2022-12-01T10:16:11.6912759Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:16:11.6913089Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:16:11.6913504Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:16:11.6913848Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:16:11.6914181Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:16:11.6914633Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:16:11.6914958Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:16:11.6915383Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:16:11.6915720Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:16:11.6916038Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:16:11.6916505Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:16:11.6916827Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:16:11.6917250Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:16:11.6917575Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:16:11.6917947Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-12-01T10:16:11.6918441Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:16:11.6918981Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:16:11.6919468Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:16:11.6919802Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:16:11.6920229Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:16:11.6920560Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:16:11.6920893Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:16:11.6921381Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:16:11.6921900Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:16:11.6922272Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:16:11.6922618Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:16:11.6922960Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:16:11.6923288Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:16:11.6923543Z dist init r=1, world=4
2022-12-01T10:16:11.6923722Z dist init r=2, world=4
2022-12-01T10:16:11.6923886Z dist init r=0, world=4
2022-12-01T10:16:11.6924058Z dist init r=3, world=4
2022-12-01T10:16:11.6924260Z skip: Need at least 2 CUDA devices (1.417s)
2022-12-01T10:16:11.6924570Z   test_error_already_wrapped_nested_False_cuda_init_mode_CUDAInitMode_CUDA_BEFORE (__main__.TestFSDPWrap)
2022-12-01T10:16:11.6924984Z Test that an error is raised if we attempt to wrap when submodules are ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12327
2022-12-01T10:16:11.6925382Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12328
2022-12-01T10:16:11.6925712Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12329
2022-12-01T10:16:11.6926445Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 12330
2022-12-01T10:16:11.6926880Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:16:11.6927212Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:16:11.6927640Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:16:11.6927975Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:16:11.6928310Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:16:11.6928780Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:16:11.6929149Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:16:11.6929560Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:16:11.6929904Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:16:11.6930236Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:16:11.6930692Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:16:11.6931056Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:16:11.6931488Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:16:11.6931866Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:16:11.6932184Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-12-01T10:16:11.6932678Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:16:11.6933171Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:16:11.6933494Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:16:11.6933906Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:16:11.6934250Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:16:11.6934581Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:16:11.6935061Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:16:11.6935575Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:16:11.6936089Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:16:11.6936472Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:16:11.6936805Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:16:11.6937141Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:16:11.6937488Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:16:11.6937839Z dist init r=0, world=4
2022-12-01T10:16:11.6938007Z dist init r=3, world=4
2022-12-01T10:16:11.6938183Z dist init r=2, world=4
2022-12-01T10:16:11.6938360Z dist init r=1, world=4
2022-12-01T10:16:11.6938552Z skip: Need at least 2 CUDA devices (1.317s)
2022-12-01T10:16:11.6938862Z   test_error_already_wrapped_nested_True_cuda_init_mode_CUDAInitMode_CUDA_AFTER (__main__.TestFSDPWrap)
2022-12-01T10:16:11.6939285Z Test that an error is raised if we attempt to wrap when submodules are ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12383
2022-12-01T10:16:11.6939667Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12384
2022-12-01T10:16:11.6939996Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12385
2022-12-01T10:16:11.6940547Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 12386
2022-12-01T10:16:11.6941019Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:16:11.6941340Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:16:11.6941770Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:16:11.6942117Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:16:11.6942440Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:16:11.6942911Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:16:11.6943239Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:16:11.6943666Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:16:11.6944079Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:16:11.6944460Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:16:11.6944924Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:16:11.6945252Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:16:11.6945661Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:16:11.6946001Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:16:11.6946332Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:16:11.6946810Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:16:11.6947332Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:16:11.6947825Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:16:11.6948148Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:16:11.6948564Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:16:11.6948904Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:16:11.6949238Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-12-01T10:16:11.6949728Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:16:11.6950227Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:16:11.6950614Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:16:11.6951043Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:16:11.6951427Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:16:11.6951768Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:16:11.6952023Z dist init r=0, world=4
2022-12-01T10:16:11.6952206Z dist init r=2, world=4
2022-12-01T10:16:11.6952370Z dist init r=3, world=4
2022-12-01T10:16:11.6952546Z dist init r=1, world=4
2022-12-01T10:16:11.6952752Z skip: Need at least 2 CUDA devices (1.417s)
2022-12-01T10:16:11.6953052Z   test_error_already_wrapped_nested_True_cuda_init_mode_CUDAInitMode_CUDA_BEFORE (__main__.TestFSDPWrap)
2022-12-01T10:16:11.6953479Z Test that an error is raised if we attempt to wrap when submodules are ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12439
2022-12-01T10:16:11.6953875Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12440
2022-12-01T10:16:11.6954208Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12441
2022-12-01T10:16:11.6954523Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 12442
2022-12-01T10:16:11.6954977Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:16:11.6955308Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:16:11.6955721Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:16:11.6956065Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:16:11.6956396Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:16:11.6956915Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:16:11.6957279Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:16:11.6957706Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:16:11.6958050Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:16:11.6958370Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:16:11.6958835Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:16:11.6959164Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:16:11.6959589Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:16:11.6959920Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:16:11.6960254Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:16:11.6960747Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:16:11.6961259Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:16:11.6961736Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:16:11.6962065Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:16:11.6962490Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:16:11.6962822Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:16:11.6963156Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-12-01T10:16:11.6963646Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:16:11.6964163Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:16:11.6964533Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:16:11.6964879Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:16:11.6965224Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:16:11.6965564Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:16:11.6965809Z dist init r=3, world=4
2022-12-01T10:16:11.6965990Z dist init r=1, world=4
2022-12-01T10:16:11.6966170Z dist init r=0, world=4
2022-12-01T10:16:11.6966335Z dist init r=2, world=4
2022-12-01T10:16:11.6966540Z skip: Need at least 2 CUDA devices (1.417s)
2022-12-01T10:16:11.6967049Z   test_main_wrap_api_cpu_offload_CPUOffload(offload_params=False)_backward_prefetch_BackwardPrefetch_BACKWARD_POST_forward_prefetch_False_cuda_init_mode_CUDAInitMode_CUDA_AFTER (__main__.TestFSDPWrap) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12495
2022-12-01T10:16:11.6967600Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12496
2022-12-01T10:16:11.6967919Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12497
2022-12-01T10:16:11.6968243Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 12498
2022-12-01T10:16:11.6968692Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:16:11.6969020Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:16:11.6969476Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:16:11.6969851Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:16:11.6970187Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:16:11.6970643Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:16:11.6970969Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:16:11.6971393Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:16:11.6971731Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:16:11.6972051Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:16:11.6972520Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:16:11.6972845Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:16:11.6973257Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:16:11.6973595Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:16:11.6973928Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-12-01T10:16:11.6974392Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:16:11.6974704Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:16:11.6975130Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:16:11.6975476Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:16:11.6975812Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:16:11.6976292Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:16:11.6976809Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:16:11.6977322Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:16:11.6977936Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:16:11.6978320Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:16:11.6978673Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:16:11.6979019Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:16:11.6979352Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:16:11.6979603Z dist init r=0, world=4
2022-12-01T10:16:11.6979786Z dist init r=1, world=4
2022-12-01T10:16:11.6979950Z dist init r=2, world=4
2022-12-01T10:16:11.6980178Z dist init r=3, world=4
2022-12-01T10:16:11.6980539Z skip: Need at least 2 CUDA devices (1.419s)
2022-12-01T10:16:11.6981057Z   test_main_wrap_api_cpu_offload_CPUOffload(offload_params=False)_backward_prefetch_BackwardPrefetch_BACKWARD_POST_forward_prefetch_False_cuda_init_mode_CUDAInitMode_CUDA_BEFORE (__main__.TestFSDPWrap) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12551
2022-12-01T10:16:11.6981581Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12552
2022-12-01T10:16:11.6981911Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12553
2022-12-01T10:16:11.6982365Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 12554
2022-12-01T10:16:11.6982870Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:16:11.6983189Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:16:11.6983618Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:16:11.6983963Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:16:11.6984285Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:16:11.6984777Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:16:11.6985106Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:16:11.6985535Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:16:11.6985868Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:16:11.6986203Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:16:11.6986667Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:16:11.6986988Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:16:11.6987414Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:16:11.6987753Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:16:11.6988084Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-12-01T10:16:11.6988579Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:16:11.6989074Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:16:11.6989404Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:16:11.6989831Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:16:11.6990163Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:16:11.6990497Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:16:11.6990984Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:16:11.6991489Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:16:11.6992006Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:16:11.6992393Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:16:11.6992736Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:16:11.6993069Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:16:11.6993406Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:16:11.6993662Z dist init r=1, world=4
2022-12-01T10:16:11.6993828Z dist init r=3, world=4
2022-12-01T10:16:11.6994008Z dist init r=2, world=4
2022-12-01T10:16:11.6994186Z dist init r=0, world=4
2022-12-01T10:16:11.6994388Z skip: Need at least 2 CUDA devices (1.419s)
2022-12-01T10:16:11.6994920Z   test_main_wrap_api_cpu_offload_CPUOffload(offload_params=False)_backward_prefetch_BackwardPrefetch_BACKWARD_POST_forward_prefetch_True_cuda_init_mode_CUDAInitMode_CUDA_AFTER (__main__.TestFSDPWrap) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12607
2022-12-01T10:16:11.6995490Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12608
2022-12-01T10:16:11.6995821Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12609
2022-12-01T10:16:11.6996133Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 12610
2022-12-01T10:16:11.6996582Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:16:11.6996914Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:16:11.6997340Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:16:11.6997672Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:16:11.6998007Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:16:11.6998472Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:16:11.6998797Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:16:11.6999211Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:16:11.6999554Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:16:11.6999883Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:16:11.7000333Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:16:11.7000654Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:16:11.7001077Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:16:11.7001420Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:16:11.7001735Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-12-01T10:16:11.7002198Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:16:11.7002524Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:16:11.7002947Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:16:11.7003275Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:16:11.7003608Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:16:11.7004103Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:16:11.7004612Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:16:11.7005131Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:16:11.7005646Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:16:11.7006033Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:16:11.7006366Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:16:11.7006703Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:16:11.7007044Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:16:11.7007301Z dist init r=3, world=4
2022-12-01T10:16:11.7007506Z dist init r=1, world=4
2022-12-01T10:16:11.7007714Z dist init r=2, world=4
2022-12-01T10:16:11.7007889Z dist init r=0, world=4
2022-12-01T10:16:11.7008078Z skip: Need at least 2 CUDA devices (1.419s)
2022-12-01T10:16:11.7008587Z   test_main_wrap_api_cpu_offload_CPUOffload(offload_params=False)_backward_prefetch_BackwardPrefetch_BACKWARD_POST_forward_prefetch_True_cuda_init_mode_CUDAInitMode_CUDA_BEFORE (__main__.TestFSDPWrap) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12663
2022-12-01T10:16:11.7009126Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12664
2022-12-01T10:16:11.7009442Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12665
2022-12-01T10:16:11.7009772Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 12666
2022-12-01T10:16:11.7010219Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:16:11.7010548Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:16:11.7010964Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:16:11.7011306Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:16:11.7011640Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:16:11.7012109Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:16:11.7012422Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:16:11.7012846Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:16:11.7013189Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:16:11.7013514Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:16:11.7013983Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:16:11.7014308Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:16:11.7014733Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:16:11.7015060Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:16:11.7015393Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-12-01T10:16:11.7015882Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:16:11.7016377Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:16:11.7016692Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:16:11.7017123Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:16:11.7017468Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:16:11.7017889Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:16:11.7018383Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:16:11.7018895Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:16:11.7019409Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:16:11.7019780Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:16:11.7020173Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:16:11.7020742Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:16:11.7021085Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:16:11.7021329Z dist init r=2, world=4
2022-12-01T10:16:11.7021508Z dist init r=0, world=4
2022-12-01T10:16:11.7021686Z dist init r=3, world=4
2022-12-01T10:16:11.7021848Z dist init r=1, world=4
2022-12-01T10:16:11.7022052Z skip: Need at least 2 CUDA devices (1.418s)
2022-12-01T10:16:11.7022554Z   test_main_wrap_api_cpu_offload_CPUOffload(offload_params=False)_backward_prefetch_BackwardPrefetch_BACKWARD_PRE_forward_prefetch_False_cuda_init_mode_CUDAInitMode_CUDA_AFTER (__main__.TestFSDPWrap) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12719
2022-12-01T10:16:11.7023076Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12720
2022-12-01T10:16:11.7023407Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12721
2022-12-01T10:16:11.7023737Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 12722
2022-12-01T10:16:11.7024196Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:16:11.7024512Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:16:11.7024937Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:16:11.7025281Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:16:11.7025614Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:16:11.7026070Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:16:11.7026397Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:16:11.7026819Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:16:11.7027150Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:16:11.7027482Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:16:11.7027950Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:16:11.7028309Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:16:11.7028719Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:16:11.7029064Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:16:11.7029398Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-12-01T10:16:11.7029879Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:16:11.7030398Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:16:11.7030892Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:16:11.7031221Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:16:11.7031641Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:16:11.7031984Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:16:11.7032318Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:16:11.7032911Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:16:11.7033462Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:16:11.7033848Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:16:11.7034198Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:16:11.7034543Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:16:11.7034873Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:16:11.7035132Z dist init r=3, world=4
2022-12-01T10:16:11.7035313Z dist init r=1, world=4
2022-12-01T10:16:11.7035478Z dist init r=2, world=4
2022-12-01T10:16:11.7035654Z dist init r=0, world=4
2022-12-01T10:16:11.7035858Z skip: Need at least 2 CUDA devices (1.318s)
2022-12-01T10:16:11.7036355Z   test_main_wrap_api_cpu_offload_CPUOffload(offload_params=False)_backward_prefetch_BackwardPrefetch_BACKWARD_PRE_forward_prefetch_False_cuda_init_mode_CUDAInitMode_CUDA_BEFORE (__main__.TestFSDPWrap) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12775
2022-12-01T10:16:11.7036886Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12776
2022-12-01T10:16:11.7037282Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12777
2022-12-01T10:16:11.7037609Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 12778
2022-12-01T10:16:11.7038045Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:16:11.7038372Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:16:11.7038797Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:16:11.7039146Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:16:11.7039470Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:16:11.7039938Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:16:11.7040262Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:16:11.7040673Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:16:11.7041014Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:16:11.7041348Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:16:11.7041806Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:16:11.7042121Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:16:11.7042540Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:16:11.7042881Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:16:11.7043198Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-12-01T10:16:11.7043689Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:16:11.7044205Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:16:11.7044696Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:16:11.7045009Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:16:11.7045467Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:16:11.7045856Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:16:11.7046192Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:16:11.7046672Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:16:11.7047189Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:16:11.7047572Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:16:11.7047921Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:16:11.7048248Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:16:11.7048584Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:16:11.7048842Z dist init r=0, world=4
2022-12-01T10:16:11.7049033Z dist init r=2, world=4
2022-12-01T10:16:11.7049210Z dist init r=3, world=4
2022-12-01T10:16:11.7049383Z dist init r=1, world=4
2022-12-01T10:16:11.7049576Z skip: Need at least 2 CUDA devices (1.418s)
2022-12-01T10:16:11.7050083Z   test_main_wrap_api_cpu_offload_CPUOffload(offload_params=False)_backward_prefetch_BackwardPrefetch_BACKWARD_PRE_forward_prefetch_True_cuda_init_mode_CUDAInitMode_CUDA_AFTER (__main__.TestFSDPWrap) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12831
2022-12-01T10:16:11.7050611Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12832
2022-12-01T10:16:11.7050941Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12833
2022-12-01T10:16:11.7051255Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 12834
2022-12-01T10:16:11.7051706Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:16:11.7052035Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:16:11.7052461Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:16:11.7052792Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:16:11.7053124Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:16:11.7053590Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:16:11.7053903Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:16:11.7054329Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:16:11.7054671Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:16:11.7055003Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:16:11.7055460Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:16:11.7055785Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:16:11.7056207Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:16:11.7056536Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:16:11.7056869Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:16:11.7057360Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:16:11.7058027Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:16:11.7058546Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:16:11.7058873Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:16:11.7059296Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:16:11.7059637Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:16:11.7059957Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-12-01T10:16:11.7060598Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:16:11.7061122Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:16:11.7061503Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:16:11.7061837Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:16:11.7062175Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:16:11.7062518Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:16:11.7062763Z dist init r=2, world=4
2022-12-01T10:16:11.7062945Z dist init r=1, world=4
2022-12-01T10:16:11.7063125Z dist init r=3, world=4
2022-12-01T10:16:11.7063287Z dist init r=0, world=4
2022-12-01T10:16:11.7063491Z skip: Need at least 2 CUDA devices (1.418s)
2022-12-01T10:16:11.7064001Z   test_main_wrap_api_cpu_offload_CPUOffload(offload_params=False)_backward_prefetch_BackwardPrefetch_BACKWARD_PRE_forward_prefetch_True_cuda_init_mode_CUDAInitMode_CUDA_BEFORE (__main__.TestFSDPWrap) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12887
2022-12-01T10:16:11.7064538Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12888
2022-12-01T10:16:11.7064858Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12889
2022-12-01T10:16:11.7065182Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 12890
2022-12-01T10:16:11.7065628Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:16:11.7065955Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:16:11.7066371Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:16:11.7066719Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:16:11.7067049Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:16:11.7067508Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:16:11.7067837Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:16:11.7068260Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:16:11.7068600Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:16:11.7068918Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:16:11.7069383Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:16:11.7069707Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:16:11.7070121Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:16:11.7070530Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:16:11.7070866Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-12-01T10:16:11.7071378Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:16:11.7071692Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:16:11.7072117Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:16:11.7072459Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:16:11.7072792Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:16:11.7073275Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:16:11.7073794Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:16:11.7074307Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:16:11.7074806Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:16:11.7075188Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:16:11.7075538Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:16:11.7075880Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:16:11.7076211Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:16:11.7076463Z dist init r=1, world=4
2022-12-01T10:16:11.7076646Z dist init r=2, world=4
2022-12-01T10:16:11.7076811Z dist init r=3, world=4
2022-12-01T10:16:11.7076986Z dist init r=0, world=4
2022-12-01T10:16:11.7077192Z skip: Need at least 2 CUDA devices (1.519s)
2022-12-01T10:16:11.7077701Z   test_main_wrap_api_cpu_offload_CPUOffload(offload_params=True)_backward_prefetch_BackwardPrefetch_BACKWARD_POST_forward_prefetch_False_cuda_init_mode_CUDAInitMode_CUDA_AFTER (__main__.TestFSDPWrap) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12943
2022-12-01T10:16:11.7078218Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12944
2022-12-01T10:16:11.7078549Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12945
2022-12-01T10:16:11.7078874Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 12946
2022-12-01T10:16:11.7079309Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:16:11.7079637Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:16:11.7080069Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:16:11.7080427Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:16:11.7080751Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:16:11.7081219Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:16:11.7081545Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:16:11.7081971Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:16:11.7082299Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:16:11.7082749Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:16:11.7083268Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:16:11.7083613Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:16:11.7084036Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:16:11.7084379Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:16:11.7084711Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-12-01T10:16:11.7085189Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:16:11.7085683Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:16:11.7086008Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:16:11.7086432Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:16:11.7086764Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:16:11.7087100Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:16:11.7087589Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:16:11.7088090Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:16:11.7088601Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:16:11.7088986Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:16:11.7089330Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:16:11.7089665Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:16:11.7090004Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:16:11.7090262Z dist init r=1, world=4
2022-12-01T10:16:11.7090429Z dist init r=0, world=4
2022-12-01T10:16:11.7090606Z dist init r=2, world=4
2022-12-01T10:16:11.7090786Z dist init r=3, world=4
2022-12-01T10:16:11.7090989Z skip: Need at least 2 CUDA devices (1.420s)
2022-12-01T10:16:11.7091484Z   test_main_wrap_api_cpu_offload_CPUOffload(offload_params=True)_backward_prefetch_BackwardPrefetch_BACKWARD_POST_forward_prefetch_False_cuda_init_mode_CUDAInitMode_CUDA_BEFORE (__main__.TestFSDPWrap) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12999
2022-12-01T10:16:11.7092015Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13000
2022-12-01T10:16:11.7092343Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13001
2022-12-01T10:16:11.7092661Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 13002
2022-12-01T10:16:11.7093115Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:16:11.7093447Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:16:11.7093874Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:16:11.7094207Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:16:11.7094541Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:16:11.7095007Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:16:11.7095334Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:16:11.7095789Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:16:11.7096166Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:16:11.7096502Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:16:11.7096958Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:16:11.7097287Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:16:11.7097799Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:16:11.7098142Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:16:11.7098463Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:16:11.7098933Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:16:11.7099257Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:16:11.7099682Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:16:11.7100009Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:16:11.7100496Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-12-01T10:16:11.7100996Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:16:11.7101501Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:16:11.7102015Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:16:11.7102531Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:16:11.7102919Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:16:11.7103250Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:16:11.7103585Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:16:11.7103928Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:16:11.7104186Z dist init r=1, world=4
2022-12-01T10:16:11.7104352Z dist init r=2, world=4
2022-12-01T10:16:11.7104529Z dist init r=3, world=4
2022-12-01T10:16:11.7104703Z dist init r=0, world=4
2022-12-01T10:16:11.7104894Z skip: Need at least 2 CUDA devices (1.419s)
2022-12-01T10:16:11.7105405Z   test_main_wrap_api_cpu_offload_CPUOffload(offload_params=True)_backward_prefetch_BackwardPrefetch_BACKWARD_POST_forward_prefetch_True_cuda_init_mode_CUDAInitMode_CUDA_AFTER (__main__.TestFSDPWrap) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13055
2022-12-01T10:16:11.7105940Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13056
2022-12-01T10:16:11.7106256Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13057
2022-12-01T10:16:11.7106583Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 13058
2022-12-01T10:16:11.7107026Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:16:11.7107350Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:16:11.7107763Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:16:11.7108106Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:16:11.7108508Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:16:11.7109016Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:16:11.7109328Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:16:11.7109754Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:16:11.7110096Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:16:11.7110414Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:16:11.7110905Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:16:11.7111232Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:16:11.7111657Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:16:11.7111986Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:16:11.7112319Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-12-01T10:16:11.7112786Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:16:11.7113098Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:16:11.7113520Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:16:11.7113858Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:16:11.7114191Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:16:11.7114669Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:16:11.7115192Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:16:11.7115705Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:16:11.7116211Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:16:11.7116581Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:16:11.7116926Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:16:11.7117269Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:16:11.7117611Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:16:11.7117857Z dist init r=2, world=4
2022-12-01T10:16:11.7118044Z dist init r=3, world=4
2022-12-01T10:16:11.7118250Z dist init r=0, world=4
2022-12-01T10:16:11.7118415Z dist init r=1, world=4
2022-12-01T10:16:11.7118618Z skip: Need at least 2 CUDA devices (1.418s)
2022-12-01T10:16:11.7119121Z   test_main_wrap_api_cpu_offload_CPUOffload(offload_params=True)_backward_prefetch_BackwardPrefetch_BACKWARD_POST_forward_prefetch_True_cuda_init_mode_CUDAInitMode_CUDA_BEFORE (__main__.TestFSDPWrap) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13111
2022-12-01T10:16:11.7119636Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13112
2022-12-01T10:16:11.7119966Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13113
2022-12-01T10:16:11.7120287Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 13114
2022-12-01T10:16:11.7120739Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:16:11.7121096Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:16:11.7121561Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:16:11.7121904Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:16:11.7122240Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:16:11.7122695Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:16:11.7123022Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:16:11.7123446Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:16:11.7123775Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:16:11.7124107Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:16:11.7124617Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:16:11.7124944Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:16:11.7125357Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:16:11.7125705Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:16:11.7126035Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-12-01T10:16:11.7126517Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:16:11.7127008Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:16:11.7127339Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:16:11.7127763Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:16:11.7128094Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:16:11.7128428Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:16:11.7128914Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:16:11.7129431Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:16:11.7129931Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:16:11.7130312Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:16:11.7130660Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:16:11.7131003Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:16:11.7131334Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:16:11.7131588Z dist init r=0, world=4
2022-12-01T10:16:11.7131767Z dist init r=1, world=4
2022-12-01T10:16:11.7131932Z dist init r=2, world=4
2022-12-01T10:16:11.7132106Z dist init r=3, world=4
2022-12-01T10:16:11.7132310Z skip: Need at least 2 CUDA devices (1.418s)
2022-12-01T10:16:11.7132804Z   test_main_wrap_api_cpu_offload_CPUOffload(offload_params=True)_backward_prefetch_BackwardPrefetch_BACKWARD_PRE_forward_prefetch_False_cuda_init_mode_CUDAInitMode_CUDA_AFTER (__main__.TestFSDPWrap) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13167
2022-12-01T10:16:11.7133332Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13168
2022-12-01T10:16:11.7133701Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13169
2022-12-01T10:16:11.7134081Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 13170
2022-12-01T10:16:11.7134516Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:16:11.7134846Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:16:11.7135271Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:16:11.7135612Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:16:11.7135933Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:16:11.7136400Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:16:11.7136726Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:16:11.7137141Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:16:11.7137483Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:16:11.7137899Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:16:11.7138366Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:16:11.7138674Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:16:11.7139097Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:16:11.7139439Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:16:11.7139759Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-12-01T10:16:11.7140226Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:16:11.7140706Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:16:11.7141136Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:16:11.7141467Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:16:11.7141803Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:16:11.7142299Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:16:11.7142816Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:16:11.7143314Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:16:11.7143831Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:16:11.7144217Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:16:11.7144563Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:16:11.7144896Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:16:11.7145236Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:16:11.7145493Z dist init r=0, world=4
2022-12-01T10:16:11.7145661Z dist init r=2, world=4
2022-12-01T10:16:11.7145835Z dist init r=1, world=4
2022-12-01T10:16:11.7146013Z dist init r=3, world=4
2022-12-01T10:16:11.7146204Z skip: Need at least 2 CUDA devices (1.419s)
2022-12-01T10:16:11.7146783Z   test_main_wrap_api_cpu_offload_CPUOffload(offload_params=True)_backward_prefetch_BackwardPrefetch_BACKWARD_PRE_forward_prefetch_False_cuda_init_mode_CUDAInitMode_CUDA_BEFORE (__main__.TestFSDPWrap) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13223
2022-12-01T10:16:11.7147360Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13224
2022-12-01T10:16:11.7147689Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13225
2022-12-01T10:16:11.7148003Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 13226
2022-12-01T10:16:11.7148452Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:16:11.7148783Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:16:11.7149213Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:16:11.7149546Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:16:11.7149883Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:16:11.7150347Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:16:11.7150661Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:16:11.7151088Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:16:11.7151432Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:16:11.7151763Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:16:11.7152217Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:16:11.7152544Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:16:11.7152972Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:16:11.7153302Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:16:11.7153630Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:16:11.7154092Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:16:11.7154414Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:16:11.7154872Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:16:11.7155211Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:16:11.7155543Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-12-01T10:16:11.7156036Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:16:11.7156544Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:16:11.7157058Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:16:11.7157570Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:16:11.7157951Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:16:11.7158282Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:16:11.7158622Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:16:11.7159001Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:16:11.7159247Z dist init r=0, world=4
2022-12-01T10:16:11.7159459Z dist init r=2, world=4
2022-12-01T10:16:11.7159638Z dist init r=3, world=4
2022-12-01T10:16:11.7159800Z dist init r=1, world=4
2022-12-01T10:16:11.7160005Z skip: Need at least 2 CUDA devices (1.419s)
2022-12-01T10:16:11.7160503Z   test_main_wrap_api_cpu_offload_CPUOffload(offload_params=True)_backward_prefetch_BackwardPrefetch_BACKWARD_PRE_forward_prefetch_True_cuda_init_mode_CUDAInitMode_CUDA_AFTER (__main__.TestFSDPWrap) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13279
2022-12-01T10:16:11.7161027Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13280
2022-12-01T10:16:11.7161343Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13281
2022-12-01T10:16:11.7161665Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 13282
2022-12-01T10:16:11.7162117Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:16:11.7162444Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:16:11.7162861Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:16:11.7163206Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:16:11.7163540Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:16:11.7163998Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:16:11.7164327Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:16:11.7164752Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:16:11.7165098Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:16:11.7165417Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:16:11.7165912Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:16:11.7166401Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:16:11.7166713Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:16:11.7167142Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:16:11.7167486Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:16:11.7167819Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:16:11.7168299Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:16:11.7168788Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:16:11.7169118Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:16:11.7169544Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:16:11.7169874Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:16:11.7170205Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-12-01T10:16:11.7170692Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:16:11.7171195Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:16:11.7171613Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:16:11.7172026Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:16:11.7172368Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:16:11.7172697Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:16:11.7172947Z dist init r=0, world=4
2022-12-01T10:16:11.7173128Z dist init r=1, world=4
2022-12-01T10:16:11.7173293Z dist init r=3, world=4
2022-12-01T10:16:11.7173471Z dist init r=2, world=4
2022-12-01T10:16:11.7173676Z skip: Need at least 2 CUDA devices (1.419s)
2022-12-01T10:16:11.7174179Z   test_main_wrap_api_cpu_offload_CPUOffload(offload_params=True)_backward_prefetch_BackwardPrefetch_BACKWARD_PRE_forward_prefetch_True_cuda_init_mode_CUDAInitMode_CUDA_BEFORE (__main__.TestFSDPWrap) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13335
2022-12-01T10:16:11.7174700Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13336
2022-12-01T10:16:11.7175034Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13337
2022-12-01T10:16:11.7175363Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 13338
2022-12-01T10:16:11.7175801Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:16:11.7176128Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:16:11.7176555Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:16:11.7176896Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:16:11.7177218Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:16:11.7177766Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:16:11.7178095Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:16:11.7178521Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:16:11.7178852Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:16:11.7179189Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:16:11.7179656Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:16:11.7179969Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:16:11.7180538Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:16:11.7180928Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:16:11.7181266Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-12-01T10:16:11.7181756Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:16:11.7182273Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:16:11.7182768Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:16:11.7183094Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:16:11.7183506Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:16:11.7183850Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:16:11.7184249Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:16:11.7184728Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:16:11.7185291Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:16:11.7185673Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:16:11.7186017Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:16:11.7186362Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:16:11.7186700Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:16:11.7186957Z dist init r=1, world=4
2022-12-01T10:16:11.7187123Z dist init r=0, world=4
2022-12-01T10:16:11.7187299Z dist init r=3, world=4
2022-12-01T10:16:11.7187474Z dist init r=2, world=4
2022-12-01T10:16:11.7187668Z skip: Need at least 2 CUDA devices (1.518s)
2022-12-01T10:16:11.7188040Z   test_wrap_batchnorm_individually_use_or_policy_False (__main__.TestFSDPWrap) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13391
2022-12-01T10:16:11.7188443Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13392
2022-12-01T10:16:11.7188774Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13393
2022-12-01T10:16:11.7189087Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 13394
2022-12-01T10:16:11.7189533Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:16:11.7189865Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:16:11.7190289Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:16:11.7190623Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:16:11.7190959Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:16:11.7191427Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:16:11.7191739Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:16:11.7192162Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:16:11.7192505Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:16:11.7192836Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:16:11.7193285Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:16:11.7193616Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:16:11.7194045Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:16:11.7194388Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:16:11.7194705Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-12-01T10:16:11.7195171Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:16:11.7195495Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:16:11.7195908Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:16:11.7196248Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:16:11.7196582Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:16:11.7197112Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:16:11.7197650Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:16:11.7198162Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:16:11.7198673Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:16:11.7199056Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:16:11.7199387Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:16:11.7199727Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:16:11.7200075Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:16:11.7200321Z dist init r=2, world=4
2022-12-01T10:16:11.7200506Z dist init r=1, world=4
2022-12-01T10:16:11.7200686Z dist init r=0, world=4
2022-12-01T10:16:11.7200850Z dist init r=3, world=4
2022-12-01T10:16:11.7201053Z skip: Need at least 2 CUDA devices (1.417s)
2022-12-01T10:16:11.7201422Z   test_wrap_batchnorm_individually_use_or_policy_True (__main__.TestFSDPWrap) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13447
2022-12-01T10:16:11.7201824Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13448
2022-12-01T10:16:11.7202138Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13449
2022-12-01T10:16:11.7202460Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 13450
2022-12-01T10:16:11.7202912Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:16:11.7203232Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:16:11.7203664Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:16:11.7204006Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:16:11.7204343Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:16:11.7204793Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:16:11.7205118Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:16:11.7205543Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:16:11.7205884Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:16:11.7206209Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:16:11.7206672Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:16:11.7206999Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:16:11.7207409Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:16:11.7207748Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:16:11.7208080Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:16:11.7208568Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:16:11.7209043Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:16:11.7209369Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:16:11.7209833Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:16:11.7210208Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:16:11.7210529Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-12-01T10:16:11.7211018Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:16:11.7211532Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:16:11.7212033Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:16:11.7212417Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:16:11.7212766Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:16:11.7213113Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:16:11.7213443Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:16:11.7213698Z dist init r=2, world=4
2022-12-01T10:16:11.7213880Z dist init r=3, world=4
2022-12-01T10:16:11.7214045Z dist init r=1, world=4
2022-12-01T10:16:11.7214219Z dist init r=0, world=4
2022-12-01T10:16:11.7214424Z skip: Need at least 2 CUDA devices (1.417s)
2022-12-01T10:16:11.7214563Z 
2022-12-01T10:16:11.7214749Z ----------------------------------------------------------------------
2022-12-01T10:16:11.7214992Z Ran 46 tests in 32.984s
2022-12-01T10:16:11.7215104Z 
2022-12-01T10:16:11.7215176Z OK (skipped=46)
2022-12-01T10:16:11.7215281Z 
2022-12-01T10:16:11.7215365Z Generating XML reports...
2022-12-01T10:16:11.7215782Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_wrap/TEST-TestAutoWrap-20221201101538.xml
2022-12-01T10:16:11.7216319Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_wrap/TEST-TestFSDPWrap-20221201101538.xml
2022-12-01T10:16:11.7216561Z 
2022-12-01T10:16:11.7216922Z ##[endgroup]
2022-12-01T10:16:11.7217337Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_wrap (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_wrap_xsqgz3g7)
2022-12-01T10:16:11.7217579Z 
2022-12-01T10:16:11.7217857Z Running distributed/fsdp/test_fsdp_comm_hooks ... [2022-12-01 10:16:11.684174]
2022-12-01T10:16:11.7218382Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_comm_hooks.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-12-01 10:16:11.684497]
2022-12-01T10:16:34.4005108Z 
2022-12-01T10:16:34.4007425Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_comm_hooks
2022-12-01T10:16:34.4022164Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_comm_hooks (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_comm_hooks_vm7g1k23)
2022-12-01T10:16:34.4023139Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_comm_hooks
2022-12-01T10:16:34.4023534Z 
2022-12-01T10:16:34.4023681Z Running tests...
2022-12-01T10:16:34.4024087Z ----------------------------------------------------------------------
2022-12-01T10:16:34.4025919Z   test_bf16_hook_has_wrapping_False_sharding_strategy_ShardingStrategy_FULL_SHARD (__main__.TestCommunicationHooks) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-12-01T10:16:34.4026763Z   test_bf16_hook_has_wrapping_False_sharding_strategy_ShardingStrategy_NO_SHARD (__main__.TestCommunicationHooks) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-12-01T10:16:34.4027589Z   test_bf16_hook_has_wrapping_False_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestCommunicationHooks) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-12-01T10:16:34.4028881Z   test_bf16_hook_has_wrapping_True_sharding_strategy_ShardingStrategy_FULL_SHARD (__main__.TestCommunicationHooks) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-12-01T10:16:34.4029780Z   test_bf16_hook_has_wrapping_True_sharding_strategy_ShardingStrategy_NO_SHARD (__main__.TestCommunicationHooks) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-12-01T10:16:34.4030678Z   test_bf16_hook_has_wrapping_True_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestCommunicationHooks) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-12-01T10:16:34.4031143Z   test_default_communication_hook_behavior_sharding_strategy_ShardingStrategy_FULL_SHARD (__main__.TestCommunicationHooks)
2022-12-01T10:16:34.4031780Z Tests FSDP's default communication hook's behavior and correctness. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13514
2022-12-01T10:16:34.4032189Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13515
2022-12-01T10:16:34.4032529Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13516
2022-12-01T10:16:34.4032846Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 13517
2022-12-01T10:16:34.4033307Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:16:34.4033646Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:16:34.4034080Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:16:34.4034419Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:16:34.4034764Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:16:34.4035237Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:16:34.4035557Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:16:34.4035991Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:16:34.4036344Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:16:34.4036680Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:16:34.4037135Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:16:34.4037466Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:16:34.4037892Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:16:34.4038242Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:16:34.4038564Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:16:34.4039036Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:16:34.4039364Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:16:34.4039779Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:16:34.4040121Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:16:34.4040459Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-12-01T10:16:34.4040960Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:16:34.4041471Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:16:34.4042037Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:16:34.4042589Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:16:34.4042981Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:16:34.4043318Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:16:34.4043662Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:16:34.4044007Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:16:34.4044257Z dist init r=2, world=4
2022-12-01T10:16:34.4044444Z dist init r=0, world=4
2022-12-01T10:16:34.4044626Z dist init r=3, world=4
2022-12-01T10:16:34.4044789Z dist init r=1, world=4
2022-12-01T10:16:34.4045000Z skip: Need at least 2 CUDA devices (1.657s)
2022-12-01T10:16:34.4045344Z   test_default_communication_hook_behavior_sharding_strategy_ShardingStrategy_NO_SHARD (__main__.TestCommunicationHooks)
2022-12-01T10:16:34.4045908Z Tests FSDP's default communication hook's behavior and correctness. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13570
2022-12-01T10:16:34.4046301Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13571
2022-12-01T10:16:34.4046635Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13572
2022-12-01T10:16:34.4046962Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 13573
2022-12-01T10:16:34.4047405Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:16:34.4047739Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:16:34.4048171Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:16:34.4048525Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:16:34.4048855Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:16:34.4049327Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:16:34.4049661Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:16:34.4050089Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:16:34.4050422Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:16:34.4050761Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:16:34.4051225Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:16:34.4051544Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:16:34.4051974Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:16:34.4052323Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:16:34.4052657Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:16:34.4053104Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:16:34.4053429Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:16:34.4053855Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:16:34.4054198Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:16:34.4054564Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-12-01T10:16:34.4055064Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:16:34.4055654Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:16:34.4056156Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:16:34.4056664Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:16:34.4057052Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:16:34.4057404Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:16:34.4057840Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:16:34.4058186Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:16:34.4058450Z dist init r=3, world=4
2022-12-01T10:16:34.4058622Z dist init r=0, world=4
2022-12-01T10:16:34.4058805Z dist init r=1, world=4
2022-12-01T10:16:34.4058984Z dist init r=2, world=4
2022-12-01T10:16:34.4059194Z skip: Need at least 2 CUDA devices (1.418s)
2022-12-01T10:16:34.4059528Z   test_default_communication_hook_behavior_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestCommunicationHooks)
2022-12-01T10:16:34.4060102Z Tests FSDP's default communication hook's behavior and correctness. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13626
2022-12-01T10:16:34.4060700Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13627
2022-12-01T10:16:34.4061023Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13628
2022-12-01T10:16:34.4061351Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 13629
2022-12-01T10:16:34.4061814Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:16:34.4062149Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:16:34.4062566Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:16:34.4062914Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:16:34.4063254Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:16:34.4063708Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:16:34.4064042Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:16:34.4064470Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:16:34.4064819Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:16:34.4065145Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:16:34.4065649Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:16:34.4066148Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:16:34.4066479Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:16:34.4066893Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:16:34.4067242Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:16:34.4067581Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:16:34.4068121Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:16:34.4068494Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:16:34.4068922Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:16:34.4069271Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:16:34.4069597Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-12-01T10:16:34.4070091Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:16:34.4070617Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:16:34.4071140Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:16:34.4071518Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:16:34.4071872Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:16:34.4072218Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:16:34.4072551Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:16:34.4072820Z dist init r=3, world=4
2022-12-01T10:16:34.4073007Z dist init r=0, world=4
2022-12-01T10:16:34.4073189Z dist init r=1, world=4
2022-12-01T10:16:34.4073358Z dist init r=2, world=4
2022-12-01T10:16:34.4073569Z skip: Need at least 2 CUDA devices (1.422s)
2022-12-01T10:16:34.4073942Z   test_default_communication_hook_initialization_has_wrapping_False_sharding_strategy_ShardingStrategy_FULL_SHARD (__main__.TestCommunicationHooks)
2022-12-01T10:16:34.4074504Z Tests FSDP's communication hook interface behavior. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13682
2022-12-01T10:16:34.4074893Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13683
2022-12-01T10:16:34.4075228Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13684
2022-12-01T10:16:34.4075560Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 13685
2022-12-01T10:16:34.4076002Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:16:34.4076333Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:16:34.4076765Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:16:34.4077098Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:16:34.4077439Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:16:34.4077912Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:16:34.4078244Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:16:34.4078656Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:16:34.4079002Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:16:34.4079361Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:16:34.4079818Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:16:34.4080148Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:16:34.4080576Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:16:34.4080965Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:16:34.4081321Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-12-01T10:16:34.4081794Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:16:34.4082125Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:16:34.4082555Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:16:34.4082888Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:16:34.4083221Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:16:34.4083715Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:16:34.4084232Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:16:34.4084767Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:16:34.4085291Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:16:34.4085680Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:16:34.4086018Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:16:34.4086357Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:16:34.4086706Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:16:34.4086965Z dist init r=3, world=4
2022-12-01T10:16:34.4087141Z dist init r=1, world=4
2022-12-01T10:16:34.4087326Z dist init r=2, world=4
2022-12-01T10:16:34.4087507Z dist init r=0, world=4
2022-12-01T10:16:34.4087706Z skip: Need at least 2 CUDA devices (1.418s)
2022-12-01T10:16:34.4088078Z   test_default_communication_hook_initialization_has_wrapping_False_sharding_strategy_ShardingStrategy_NO_SHARD (__main__.TestCommunicationHooks)
2022-12-01T10:16:34.4088649Z Tests FSDP's communication hook interface behavior. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13738
2022-12-01T10:16:34.4089023Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13739
2022-12-01T10:16:34.4089355Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13740
2022-12-01T10:16:34.4089682Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 13741
2022-12-01T10:16:34.4090138Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:16:34.4090467Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:16:34.4090902Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:16:34.4091250Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:16:34.4091573Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:16:34.4092046Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:16:34.4092375Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:16:34.4092803Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:16:34.4093135Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:16:34.4093514Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:16:34.4093992Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:16:34.4094354Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:16:34.4094770Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:16:34.4095129Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:16:34.4095470Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-12-01T10:16:34.4095961Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:16:34.4096465Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:16:34.4096800Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:16:34.4097236Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:16:34.4097657Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:16:34.4097998Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:16:34.4098500Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:16:34.4099028Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:16:34.4099540Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:16:34.4099935Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:16:34.4100287Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:16:34.4100847Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:16:34.4101195Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:16:34.4101452Z dist init r=3, world=4
2022-12-01T10:16:34.4101645Z dist init r=2, world=4
2022-12-01T10:16:34.4101812Z dist init r=1, world=4
2022-12-01T10:16:34.4101997Z dist init r=0, world=4
2022-12-01T10:16:34.4102209Z skip: Need at least 2 CUDA devices (1.418s)
2022-12-01T10:16:34.4102575Z   test_default_communication_hook_initialization_has_wrapping_False_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestCommunicationHooks)
2022-12-01T10:16:34.4103157Z Tests FSDP's communication hook interface behavior. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13794
2022-12-01T10:16:34.4103550Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13795
2022-12-01T10:16:34.4103893Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13796
2022-12-01T10:16:34.4104213Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 13797
2022-12-01T10:16:34.4104673Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:16:34.4105010Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:16:34.4105434Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:16:34.4105784Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:16:34.4106129Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:16:34.4106599Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:16:34.4107006Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:16:34.4107481Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:16:34.4107827Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:16:34.4108150Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:16:34.4108646Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:16:34.4109149Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:16:34.4109481Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:16:34.4109903Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:16:34.4110254Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:16:34.4110593Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:16:34.4111089Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:16:34.4111573Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:16:34.4111903Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:16:34.4112334Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:16:34.4112669Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:16:34.4113012Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-12-01T10:16:34.4113513Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:16:34.4114038Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:16:34.4114413Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:16:34.4114768Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:16:34.4115116Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:16:34.4115461Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:16:34.4115706Z dist init r=1, world=4
2022-12-01T10:16:34.4115892Z dist init r=0, world=4
2022-12-01T10:16:34.4116073Z dist init r=3, world=4
2022-12-01T10:16:34.4116239Z dist init r=2, world=4
2022-12-01T10:16:34.4116452Z skip: Need at least 2 CUDA devices (1.418s)
2022-12-01T10:16:34.4116826Z   test_default_communication_hook_initialization_has_wrapping_True_sharding_strategy_ShardingStrategy_FULL_SHARD (__main__.TestCommunicationHooks)
2022-12-01T10:16:34.4117379Z Tests FSDP's communication hook interface behavior. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13850
2022-12-01T10:16:34.4117773Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13851
2022-12-01T10:16:34.4118107Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13852
2022-12-01T10:16:34.4118439Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 13853
2022-12-01T10:16:34.4118879Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:16:34.4119213Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:16:34.4119648Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:16:34.4120043Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:16:34.4120406Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:16:34.4120876Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:16:34.4121205Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:16:34.4121622Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:16:34.4121969Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:16:34.4122307Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:16:34.4122783Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:16:34.4123102Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:16:34.4123535Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:16:34.4123884Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:16:34.4124208Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:16:34.4124674Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:16:34.4125006Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:16:34.4125432Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:16:34.4125763Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:16:34.4126100Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-12-01T10:16:34.4126591Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:16:34.4127114Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:16:34.4127622Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:16:34.4128139Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:16:34.4128525Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:16:34.4128863Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:16:34.4129211Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:16:34.4129554Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:16:34.4129820Z dist init r=1, world=4
2022-12-01T10:16:34.4129989Z dist init r=0, world=4
2022-12-01T10:16:34.4130170Z dist init r=3, world=4
2022-12-01T10:16:34.4130354Z dist init r=2, world=4
2022-12-01T10:16:34.4130548Z skip: Need at least 2 CUDA devices (1.318s)
2022-12-01T10:16:34.4130916Z   test_default_communication_hook_initialization_has_wrapping_True_sharding_strategy_ShardingStrategy_NO_SHARD (__main__.TestCommunicationHooks)
2022-12-01T10:16:34.4131485Z Tests FSDP's communication hook interface behavior. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13906
2022-12-01T10:16:34.4131872Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13907
2022-12-01T10:16:34.4132187Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13908
2022-12-01T10:16:34.4132546Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 13909
2022-12-01T10:16:34.4132996Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:16:34.4133351Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:16:34.4133781Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:16:34.4134130Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:16:34.4134466Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:16:34.4134921Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:16:34.4135248Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:16:34.4135674Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:16:34.4136008Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:16:34.4136347Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:16:34.4136816Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:16:34.4137142Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:16:34.4137635Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:16:34.4137986Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:16:34.4138326Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-12-01T10:16:34.4138828Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:16:34.4139316Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:16:34.4139652Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:16:34.4140085Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:16:34.4140562Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:16:34.4140907Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:16:34.4141411Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:16:34.4141933Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:16:34.4142443Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:16:34.4142828Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:16:34.4143178Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:16:34.4143526Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:16:34.4143859Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:16:34.4144113Z dist init r=1, world=4
2022-12-01T10:16:34.4144298Z dist init r=0, world=4
2022-12-01T10:16:34.4144465Z dist init r=3, world=4
2022-12-01T10:16:34.4144645Z dist init r=2, world=4
2022-12-01T10:16:34.4144855Z skip: Need at least 2 CUDA devices (1.418s)
2022-12-01T10:16:34.4145217Z   test_default_communication_hook_initialization_has_wrapping_True_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestCommunicationHooks)
2022-12-01T10:16:34.4145856Z Tests FSDP's communication hook interface behavior. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13962
2022-12-01T10:16:34.4146289Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13963
2022-12-01T10:16:34.4146619Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13964
2022-12-01T10:16:34.4146931Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 13965
2022-12-01T10:16:34.4147387Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:16:34.4147721Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:16:34.4148161Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:16:34.4148495Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:16:34.4148841Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:16:34.4149313Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:16:34.4149628Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:16:34.4150054Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:16:34.4150400Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:16:34.4150736Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:16:34.4151193Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:16:34.4151526Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:16:34.4151956Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:16:34.4152288Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:16:34.4152628Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:16:34.4153095Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:16:34.4153423Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:16:34.4153836Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:16:34.4154180Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:16:34.4154515Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-12-01T10:16:34.4155012Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:16:34.4155527Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:16:34.4156048Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:16:34.4156557Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:16:34.4156944Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:16:34.4157295Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:16:34.4157645Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:16:34.4157977Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:16:34.4158233Z dist init r=3, world=4
2022-12-01T10:16:34.4158417Z dist init r=0, world=4
2022-12-01T10:16:34.4158620Z dist init r=2, world=4
2022-12-01T10:16:34.4158799Z dist init r=1, world=4
2022-12-01T10:16:34.4159038Z skip: Need at least 2 CUDA devices (1.418s)
2022-12-01T10:16:34.4159415Z   test_fp16_hook_has_wrapping_False_sharding_strategy_ShardingStrategy_FULL_SHARD (__main__.TestCommunicationHooks) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-12-01T10:16:34.4159932Z   test_fp16_hook_has_wrapping_False_sharding_strategy_ShardingStrategy_NO_SHARD (__main__.TestCommunicationHooks) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-12-01T10:16:34.4160445Z   test_fp16_hook_has_wrapping_False_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestCommunicationHooks) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2022-12-01T10:16:34.4160957Z   test_fp16_hook_has_wrapping_True_sharding_strategy_ShardingStrategy_FULL_SHARD (__main__.TestCommunicationHooks) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2022-12-01T10:16:34.4161462Z   test_fp16_hook_has_wrapping_True_sharding_strategy_ShardingStrategy_NO_SHARD (__main__.TestCommunicationHooks) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2022-12-01T10:16:34.4161959Z   test_fp16_hook_has_wrapping_True_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestCommunicationHooks) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2022-12-01T10:16:34.4162416Z   test_registering_hook_non_root_sharding_strategy_ShardingStrategy_FULL_SHARD (__main__.TestCommunicationHooks)
2022-12-01T10:16:34.4162952Z Tests FSDP's communication hook registering for submodules. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14018
2022-12-01T10:16:34.4163342Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14019
2022-12-01T10:16:34.4163657Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14020
2022-12-01T10:16:34.4163979Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 14021
2022-12-01T10:16:34.4174498Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:16:34.4174875Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:16:34.4175346Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:16:34.4175792Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:16:34.4176153Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:16:34.4176658Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:16:34.4177001Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:16:34.4177463Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:16:34.4177920Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:16:34.4178266Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:16:34.4178738Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:16:34.4179076Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:16:34.4179510Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:16:34.4179859Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:16:34.4180183Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:16:34.4180814Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:16:34.4181276Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:16:34.4181744Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:16:34.4182094Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:16:34.4182430Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-12-01T10:16:34.4182926Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:16:34.4183432Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:16:34.4183942Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:16:34.4184451Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:16:34.4184835Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:16:34.4185170Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:16:34.4185509Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:16:34.4185852Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:16:34.4186096Z dist init r=2, world=4
2022-12-01T10:16:34.4186280Z dist init r=1, world=4
2022-12-01T10:16:34.4186461Z dist init r=0, world=4
2022-12-01T10:16:34.4186624Z dist init r=3, world=4
2022-12-01T10:16:34.4186832Z skip: Need at least 2 CUDA devices (1.417s)
2022-12-01T10:16:34.4187157Z   test_registering_hook_non_root_sharding_strategy_ShardingStrategy_NO_SHARD (__main__.TestCommunicationHooks)
2022-12-01T10:16:34.4187695Z Tests FSDP's communication hook registering for submodules. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14074
2022-12-01T10:16:34.4188077Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14075
2022-12-01T10:16:34.4188408Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14076
2022-12-01T10:16:34.4188735Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 14077
2022-12-01T10:16:34.4189171Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:16:34.4189503Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:16:34.4189930Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:16:34.4190273Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:16:34.4190594Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:16:34.4191066Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:16:34.4191396Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:16:34.4191819Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:16:34.4192148Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:16:34.4192483Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:16:34.4192947Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:16:34.4193261Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:16:34.4193685Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:16:34.4194082Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:16:34.4194452Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-12-01T10:16:34.4194906Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:16:34.4195234Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:16:34.4195662Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:16:34.4196005Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:16:34.4196327Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:16:34.4196821Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:16:34.4197343Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:16:34.4197850Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:16:34.4198365Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:16:34.4198753Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:16:34.4199101Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:16:34.4199436Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:16:34.4199773Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:16:34.4200035Z dist init r=3, world=4
2022-12-01T10:16:34.4200204Z dist init r=1, world=4
2022-12-01T10:16:34.4200383Z dist init r=2, world=4
2022-12-01T10:16:34.4200561Z dist init r=0, world=4
2022-12-01T10:16:34.4200769Z skip: Need at least 2 CUDA devices (1.417s)
2022-12-01T10:16:34.4201087Z   test_registering_hook_non_root_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestCommunicationHooks)
2022-12-01T10:16:34.4201628Z Tests FSDP's communication hook registering for submodules. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14130
2022-12-01T10:16:34.4202017Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14131
2022-12-01T10:16:34.4202336Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14132
2022-12-01T10:16:34.4202657Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 14133
2022-12-01T10:16:34.4203109Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:16:34.4203440Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:16:34.4203858Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:16:34.4204204Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:16:34.4204538Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:16:34.4204997Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:16:34.4205327Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:16:34.4205752Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:16:34.4206093Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:16:34.4206415Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:16:34.4206924Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:16:34.4207282Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:16:34.4207708Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:16:34.4208038Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:16:34.4208369Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-12-01T10:16:34.4208862Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:16:34.4209369Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:16:34.4209866Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:16:34.4210325Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:16:34.4210763Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:16:34.4211094Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:16:34.4211433Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:16:34.4211927Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:16:34.4212444Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:16:34.4212817Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:16:34.4213168Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:16:34.4213517Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:16:34.4213850Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:16:34.4214111Z dist init r=0, world=4
2022-12-01T10:16:34.4214297Z dist init r=1, world=4
2022-12-01T10:16:34.4214474Z dist init r=2, world=4
2022-12-01T10:16:34.4214640Z dist init r=3, world=4
2022-12-01T10:16:34.4214843Z skip: Need at least 2 CUDA devices (1.419s)
2022-12-01T10:16:34.4215176Z   test_registering_hook_submodules_sharding_strategy_ShardingStrategy_FULL_SHARD (__main__.TestCommunicationHooks)
2022-12-01T10:16:34.4215707Z Tests FSDP's communication hook registering for submodules. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14186
2022-12-01T10:16:34.4216097Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14187
2022-12-01T10:16:34.4216429Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14188
2022-12-01T10:16:34.4216743Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 14189
2022-12-01T10:16:34.4217195Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:16:34.4217605Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:16:34.4218049Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:16:34.4218385Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:16:34.4218724Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:16:34.4219193Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:16:34.4219521Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:16:34.4219988Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:16:34.4220479Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:16:34.4220838Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:16:34.4221326Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:16:34.4221821Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:16:34.4222148Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:16:34.4222577Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:16:34.4222909Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:16:34.4223249Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:16:34.4223743Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:16:34.4224236Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:16:34.4224549Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:16:34.4224974Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:16:34.4225315Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:16:34.4225638Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-12-01T10:16:34.4226129Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:16:34.4226650Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:16:34.4227038Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:16:34.4227374Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:16:34.4227713Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:16:34.4228059Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:16:34.4228315Z dist init r=1, world=4
2022-12-01T10:16:34.4228484Z dist init r=3, world=4
2022-12-01T10:16:34.4228661Z dist init r=0, world=4
2022-12-01T10:16:34.4228835Z dist init r=2, world=4
2022-12-01T10:16:34.4229027Z skip: Need at least 2 CUDA devices (1.417s)
2022-12-01T10:16:34.4229359Z   test_registering_hook_submodules_sharding_strategy_ShardingStrategy_NO_SHARD (__main__.TestCommunicationHooks)
2022-12-01T10:16:34.4229896Z Tests FSDP's communication hook registering for submodules. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14242
2022-12-01T10:16:34.4230277Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14243
2022-12-01T10:16:34.4230606Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14244
2022-12-01T10:16:34.4230936Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 14245
2022-12-01T10:16:34.4231380Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:16:34.4231693Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:16:34.4232118Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:16:34.4232462Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:16:34.4232850Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:16:34.4233370Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:16:34.4233701Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:16:34.4234131Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:16:34.4234463Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:16:34.4234799Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:16:34.4235266Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:16:34.4235596Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:16:34.4236016Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:16:34.4236364Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:16:34.4236697Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:16:34.4237150Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:16:34.4237473Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:16:34.4237903Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:16:34.4238244Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:16:34.4238564Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-12-01T10:16:34.4239060Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:16:34.4239582Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:16:34.4240095Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:16:34.4240594Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:16:34.4240981Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:16:34.4241329Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:16:34.4241665Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:16:34.4242003Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:16:34.4242259Z dist init r=1, world=4
2022-12-01T10:16:34.4242442Z dist init r=3, world=4
2022-12-01T10:16:34.4242611Z dist init r=0, world=4
2022-12-01T10:16:34.4242789Z dist init r=2, world=4
2022-12-01T10:16:34.4242993Z skip: Need at least 2 CUDA devices (1.417s)
2022-12-01T10:16:34.4243316Z   test_registering_hook_submodules_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestCommunicationHooks)
2022-12-01T10:16:34.4243863Z Tests FSDP's communication hook registering for submodules. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14298
2022-12-01T10:16:34.4244256Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14299
2022-12-01T10:16:34.4244570Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14300
2022-12-01T10:16:34.4244901Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 14301
2022-12-01T10:16:34.4245386Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:16:34.4245719Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:16:34.4246170Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:16:34.4246515Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:16:34.4246850Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:16:34.4247319Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:16:34.4247636Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:16:34.4248067Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:16:34.4248407Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:16:34.4248731Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:16:34.4249199Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:16:34.4249524Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:16:34.4249947Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:16:34.4250271Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:16:34.4250605Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-12-01T10:16:34.4251070Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:16:34.4251384Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:16:34.4251812Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:16:34.4252162Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:16:34.4252496Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:16:34.4252974Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:16:34.4253494Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:16:34.4254009Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:16:34.4254523Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:16:34.4254896Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:16:34.4255244Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:16:34.4255592Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:16:34.4255935Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:16:34.4256181Z dist init r=0, world=4
2022-12-01T10:16:34.4256363Z dist init r=2, world=4
2022-12-01T10:16:34.4256539Z dist init r=1, world=4
2022-12-01T10:16:34.4256702Z dist init r=3, world=4
2022-12-01T10:16:34.4256906Z skip: Need at least 2 CUDA devices (1.417s)
2022-12-01T10:16:34.4257043Z 
2022-12-01T10:16:34.4257245Z ----------------------------------------------------------------------
2022-12-01T10:16:34.4257476Z Ran 27 tests in 21.418s
2022-12-01T10:16:34.4257678Z 
2022-12-01T10:16:34.4257754Z OK (skipped=27)
2022-12-01T10:16:34.4257863Z 
2022-12-01T10:16:34.4257949Z Generating XML reports...
2022-12-01T10:16:34.4258457Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_comm_hooks/TEST-TestCommunicationHooks-20221201101612.xml
2022-12-01T10:16:34.4258766Z 
2022-12-01T10:16:34.4259222Z ##[endgroup]
2022-12-01T10:16:34.4259692Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_comm_hooks (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_comm_hooks_vm7g1k23)
2022-12-01T10:16:34.4259961Z 
2022-12-01T10:16:34.4260162Z Running distributed/fsdp/test_fsdp_grad_acc ... [2022-12-01 10:16:34.401415]
2022-12-01T10:16:34.4260838Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_grad_acc.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-12-01 10:16:34.401761]
2022-12-01T10:16:53.1463510Z 
2022-12-01T10:16:53.1464060Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_grad_acc
2022-12-01T10:16:53.1465138Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_grad_acc (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_grad_acc_aafxs4_0)
2022-12-01T10:16:53.1476381Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_grad_acc
2022-12-01T10:16:53.1476707Z 
2022-12-01T10:16:53.1476788Z Running tests...
2022-12-01T10:16:53.1477109Z ----------------------------------------------------------------------
2022-12-01T10:16:53.1477631Z   test_grad_acc_configs_[(use_no_sync=False,num_iters=3),(use_no_sync=True,num_iters=3),(use_no_sync=False,num_iters=3)]_cpu_offload_CPUOffload(offload_params=False)_sharding_strategy_ShardingStrategy_FULL_SHARD (__main__.TestGradAcc)
2022-12-01T10:16:53.1479004Z Tests gradient accumulation. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14365
2022-12-01T10:16:53.1479610Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14366
2022-12-01T10:16:53.1480120Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14367
2022-12-01T10:16:53.1480478Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 14368
2022-12-01T10:16:53.1481173Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:16:53.1481767Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:16:53.1482352Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:16:53.1482705Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:16:53.1490839Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:16:53.1491762Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:16:53.1492301Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:16:53.1493012Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:16:53.1493576Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:16:53.1494100Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:16:53.1494872Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:16:53.1495361Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:16:53.1496002Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:16:53.1496569Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:16:53.1497133Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-12-01T10:16:53.1498010Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:16:53.1498781Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:16:53.1499682Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:16:53.1500315Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:16:53.1501120Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:16:53.1502051Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:16:53.1502993Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:16:53.1503913Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:16:53.1504884Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:16:53.1505503Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:16:53.1506085Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:16:53.1506681Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:16:53.1507210Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:16:53.1507631Z dist init r=3, world=4
2022-12-01T10:16:53.1507908Z dist init r=1, world=4
2022-12-01T10:16:53.1508170Z dist init r=2, world=4
2022-12-01T10:16:53.1508453Z dist init r=0, world=4
2022-12-01T10:16:53.1508793Z skip: Need at least 2 CUDA devices (1.654s)
2022-12-01T10:16:53.1509460Z   test_grad_acc_configs_[(use_no_sync=False,num_iters=3),(use_no_sync=True,num_iters=3),(use_no_sync=False,num_iters=3)]_cpu_offload_CPUOffload(offload_params=False)_sharding_strategy_ShardingStrategy_NO_SHARD (__main__.TestGradAcc)
2022-12-01T10:16:53.1510292Z Tests gradient accumulation. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14421
2022-12-01T10:16:53.1510865Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14422
2022-12-01T10:16:53.1511346Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14423
2022-12-01T10:16:53.1511844Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 14424
2022-12-01T10:16:53.1512558Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:16:53.1517489Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:16:53.1518383Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:16:53.1519055Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:16:53.1519689Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:16:53.1520216Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:16:53.1520554Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:16:53.1520996Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:16:53.1521347Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:16:53.1521673Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:16:53.1522149Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:16:53.1522484Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:16:53.1523041Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:16:53.1523433Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:16:53.1523772Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:16:53.1524247Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:16:53.1524567Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:16:53.1524995Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:16:53.1525343Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:16:53.1525678Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-12-01T10:16:53.1526159Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:16:53.1526681Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:16:53.1529892Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:16:53.1530748Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:16:53.1531509Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:16:53.1531933Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:16:53.1532286Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:16:53.1532622Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:16:53.1532950Z dist init r=2, world=4
2022-12-01T10:16:53.1533182Z dist init r=3, world=4
2022-12-01T10:16:53.1533451Z dist init r=1, world=4
2022-12-01T10:16:53.1533751Z dist init r=0, world=4
2022-12-01T10:16:53.1534086Z skip: Need at least 2 CUDA devices (1.417s)
2022-12-01T10:16:53.1534882Z   test_grad_acc_configs_[(use_no_sync=False,num_iters=3),(use_no_sync=True,num_iters=3),(use_no_sync=False,num_iters=3)]_cpu_offload_CPUOffload(offload_params=False)_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestGradAcc)
2022-12-01T10:16:53.1535843Z Tests gradient accumulation. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14477
2022-12-01T10:16:53.1536517Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14478
2022-12-01T10:16:53.1537128Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14479
2022-12-01T10:16:53.1537638Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 14480
2022-12-01T10:16:53.1538208Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:16:53.1538549Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:16:53.1538983Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:16:53.1539331Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:16:53.1539658Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:16:53.1540136Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:16:53.1540664Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:16:53.1541213Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:16:53.1541905Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:16:53.1542510Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:16:53.1543274Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:16:53.1543863Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:16:53.1544667Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:16:53.1545283Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:16:53.1545891Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:16:53.1546441Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:16:53.1546944Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:16:53.1547279Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:16:53.1547697Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:16:53.1548043Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:16:53.1548535Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-12-01T10:16:53.1549360Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:16:53.1550204Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:16:53.1551175Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:16:53.1551945Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:16:53.1552538Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:16:53.1552876Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:16:53.1553213Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:16:53.1553476Z dist init r=3, world=4
2022-12-01T10:16:53.1553648Z dist init r=0, world=4
2022-12-01T10:16:53.1553829Z dist init r=2, world=4
2022-12-01T10:16:53.1554009Z dist init r=1, world=4
2022-12-01T10:16:53.1554207Z skip: Need at least 2 CUDA devices (1.417s)
2022-12-01T10:16:53.1554645Z   test_grad_acc_configs_[(use_no_sync=False,num_iters=3),(use_no_sync=True,num_iters=3),(use_no_sync=False,num_iters=3)]_cpu_offload_CPUOffload(offload_params=True)_sharding_strategy_ShardingStrategy_FULL_SHARD (__main__.TestGradAcc)
2022-12-01T10:16:53.1555161Z Tests gradient accumulation. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14533
2022-12-01T10:16:53.1555533Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14534
2022-12-01T10:16:53.1555852Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14535
2022-12-01T10:16:53.1556320Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 14536
2022-12-01T10:16:53.1556787Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:16:53.1557122Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:16:53.1557543Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:16:53.1557891Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:16:53.1558312Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:16:53.1558775Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:16:53.1559151Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:16:53.1559582Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:16:53.1559970Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:16:53.1560302Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:16:53.1560774Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:16:53.1561104Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:16:53.1561538Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:16:53.1561869Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:16:53.1562211Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:16:53.1562682Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:16:53.1562997Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:16:53.1563427Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:16:53.1563784Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:16:53.1564121Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-12-01T10:16:53.1564608Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:16:53.1565137Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:16:53.1565662Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:16:53.1566181Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:16:53.1566551Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:16:53.1566898Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:16:53.1567245Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:16:53.1567580Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:16:53.1567845Z dist init r=1, world=4
2022-12-01T10:16:53.1568036Z dist init r=0, world=4
2022-12-01T10:16:53.1568219Z dist init r=3, world=4
2022-12-01T10:16:53.1568387Z dist init r=2, world=4
2022-12-01T10:16:53.1568599Z skip: Need at least 2 CUDA devices (1.417s)
2022-12-01T10:16:53.1569032Z   test_grad_acc_configs_[(use_no_sync=False,num_iters=3),(use_no_sync=True,num_iters=3),(use_no_sync=False,num_iters=3)]_cpu_offload_CPUOffload(offload_params=True)_sharding_strategy_ShardingStrategy_NO_SHARD (__main__.TestGradAcc)
2022-12-01T10:16:53.1569523Z Tests gradient accumulation. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14589
2022-12-01T10:16:53.1569885Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14590
2022-12-01T10:16:53.1570217Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14591
2022-12-01T10:16:53.1570548Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 14592
2022-12-01T10:16:53.1571036Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:16:53.1571396Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:16:53.1571832Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:16:53.1572170Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:16:53.1572514Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:16:53.1572991Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:16:53.1573324Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:16:53.1573747Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:16:53.1574100Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:16:53.1574439Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:16:53.1574921Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:16:53.1575237Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:16:53.1575672Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:16:53.1576022Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:16:53.1576345Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-12-01T10:16:53.1576814Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:16:53.1577145Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:16:53.1577677Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:16:53.1578017Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:16:53.1578357Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:16:53.1578859Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:16:53.1579374Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:16:53.1579902Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:16:53.1580567Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:16:53.1580971Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:16:53.1581310Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:16:53.1581664Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:16:53.1582008Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:16:53.1582273Z dist init r=1, world=4
2022-12-01T10:16:53.1582444Z dist init r=2, world=4
2022-12-01T10:16:53.1582625Z dist init r=3, world=4
2022-12-01T10:16:53.1582815Z dist init r=0, world=4
2022-12-01T10:16:53.1583017Z skip: Need at least 2 CUDA devices (1.417s)
2022-12-01T10:16:53.1583459Z   test_grad_acc_configs_[(use_no_sync=False,num_iters=3),(use_no_sync=True,num_iters=3),(use_no_sync=False,num_iters=3)]_cpu_offload_CPUOffload(offload_params=True)_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestGradAcc)
2022-12-01T10:16:53.1584051Z Tests gradient accumulation. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14645
2022-12-01T10:16:53.1584464Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14646
2022-12-01T10:16:53.1584784Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14647
2022-12-01T10:16:53.1585115Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 14648
2022-12-01T10:16:53.1585583Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:16:53.1585906Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:16:53.1586340Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:16:53.1586690Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:16:53.1587027Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:16:53.1587486Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:16:53.1587813Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:16:53.1588243Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:16:53.1588575Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:16:53.1588914Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:16:53.1589385Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:16:53.1589715Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:16:53.1590131Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:16:53.1590487Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:16:53.1590827Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-12-01T10:16:53.1591341Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:16:53.1591829Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:16:53.1592158Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:16:53.1592596Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:16:53.1592929Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:16:53.1593267Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:16:53.1593773Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:16:53.1594301Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:16:53.1594806Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:16:53.1595196Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:16:53.1595546Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:16:53.1595896Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:16:53.1596227Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:16:53.1596483Z dist init r=0, world=4
2022-12-01T10:16:53.1596672Z dist init r=1, world=4
2022-12-01T10:16:53.1596842Z dist init r=3, world=4
2022-12-01T10:16:53.1597074Z dist init r=2, world=4
2022-12-01T10:16:53.1597283Z skip: Need at least 2 CUDA devices (1.417s)
2022-12-01T10:16:53.1597764Z   test_grad_acc_configs_[(use_no_sync=True,num_iters=3),(use_no_sync=False,num_iters=3),(use_no_sync=True,num_iters=3)]_cpu_offload_CPUOffload(offload_params=False)_sharding_strategy_ShardingStrategy_FULL_SHARD (__main__.TestGradAcc)
2022-12-01T10:16:53.1598269Z Tests gradient accumulation. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14701
2022-12-01T10:16:53.1598642Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14702
2022-12-01T10:16:53.1598976Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14703
2022-12-01T10:16:53.1599291Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 14704
2022-12-01T10:16:53.1599748Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:16:53.1600084Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:16:53.1600524Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:16:53.1600861Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:16:53.1601198Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:16:53.1601668Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:16:53.1601986Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:16:53.1602415Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:16:53.1602761Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:16:53.1603100Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:16:53.1603552Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:16:53.1603886Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:16:53.1604319Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:16:53.1604662Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:16:53.1604985Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-12-01T10:16:53.1605458Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:16:53.1605787Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:16:53.1606204Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:16:53.1606552Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:16:53.1606892Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:16:53.1607390Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:16:53.1607896Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:16:53.1608418Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:16:53.1608932Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:16:53.1609319Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:16:53.1609699Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:16:53.1610076Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:16:53.1610421Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:16:53.1610667Z dist init r=1, world=4
2022-12-01T10:16:53.1610852Z dist init r=0, world=4
2022-12-01T10:16:53.1611037Z dist init r=2, world=4
2022-12-01T10:16:53.1611204Z dist init r=3, world=4
2022-12-01T10:16:53.1611411Z skip: Need at least 2 CUDA devices (1.417s)
2022-12-01T10:16:53.1611844Z   test_grad_acc_configs_[(use_no_sync=True,num_iters=3),(use_no_sync=False,num_iters=3),(use_no_sync=True,num_iters=3)]_cpu_offload_CPUOffload(offload_params=False)_sharding_strategy_ShardingStrategy_NO_SHARD (__main__.TestGradAcc)
2022-12-01T10:16:53.1612344Z Tests gradient accumulation. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14757
2022-12-01T10:16:53.1612704Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14758
2022-12-01T10:16:53.1613039Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14759
2022-12-01T10:16:53.1613364Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 14760
2022-12-01T10:16:53.1613823Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:16:53.1614147Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:16:53.1614579Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:16:53.1614929Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:16:53.1615254Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:16:53.1615727Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:16:53.1616061Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:16:53.1616490Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:16:53.1616822Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:16:53.1617154Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:16:53.1617705Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:16:53.1618022Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:16:53.1618451Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:16:53.1618796Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:16:53.1619132Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-12-01T10:16:53.1619589Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:16:53.1619917Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:16:53.1620340Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:16:53.1620818Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:16:53.1621143Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:16:53.1621646Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:16:53.1622167Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:16:53.1622739Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:16:53.1623300Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:16:53.1623693Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:16:53.1624045Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:16:53.1624379Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:16:53.1624716Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:16:53.1624985Z dist init r=2, world=4
2022-12-01T10:16:53.1625170Z dist init r=0, world=4
2022-12-01T10:16:53.1625337Z dist init r=1, world=4
2022-12-01T10:16:53.1625516Z dist init r=3, world=4
2022-12-01T10:16:53.1625724Z skip: Need at least 2 CUDA devices (1.517s)
2022-12-01T10:16:53.1626149Z   test_grad_acc_configs_[(use_no_sync=True,num_iters=3),(use_no_sync=False,num_iters=3),(use_no_sync=True,num_iters=3)]_cpu_offload_CPUOffload(offload_params=False)_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestGradAcc)
2022-12-01T10:16:53.1626658Z Tests gradient accumulation. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14813
2022-12-01T10:16:53.1627025Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14814
2022-12-01T10:16:53.1627360Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14815
2022-12-01T10:16:53.1627676Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 14816
2022-12-01T10:16:53.1628129Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:16:53.1628457Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:16:53.1628876Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:16:53.1629224Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:16:53.1629561Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:16:53.1630031Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:16:53.1630343Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:16:53.1630774Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:16:53.1631118Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:16:53.1631443Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:16:53.1631912Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:16:53.1632241Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:16:53.1632664Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:16:53.1632993Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:16:53.1633332Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-12-01T10:16:53.1633800Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:16:53.1634127Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:16:53.1634537Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:16:53.1634915Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:16:53.1635255Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:16:53.1635778Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:16:53.1636301Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:16:53.1636817Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:16:53.1637338Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:16:53.1637708Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:16:53.1638057Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:16:53.1638405Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:16:53.1638751Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:16:53.1638999Z dist init r=0, world=4
2022-12-01T10:16:53.1639181Z dist init r=2, world=4
2022-12-01T10:16:53.1639362Z dist init r=1, world=4
2022-12-01T10:16:53.1639527Z dist init r=3, world=4
2022-12-01T10:16:53.1639739Z skip: Need at least 2 CUDA devices (1.417s)
2022-12-01T10:16:53.1640176Z   test_grad_acc_configs_[(use_no_sync=True,num_iters=3),(use_no_sync=False,num_iters=3),(use_no_sync=True,num_iters=3)]_cpu_offload_CPUOffload(offload_params=True)_sharding_strategy_ShardingStrategy_FULL_SHARD (__main__.TestGradAcc)
2022-12-01T10:16:53.1640668Z Tests gradient accumulation. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14869
2022-12-01T10:16:53.1641033Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14870
2022-12-01T10:16:53.1641368Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14871
2022-12-01T10:16:53.1641703Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 14872
2022-12-01T10:16:53.1642145Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:16:53.1642476Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:16:53.1642903Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:16:53.1643249Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:16:53.1643577Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:16:53.1644051Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:16:53.1644383Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:16:53.1644797Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:16:53.1645142Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:16:53.1645481Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:16:53.1645942Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:16:53.1646255Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:16:53.1646682Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:16:53.1647030Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:16:53.1647386Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:16:53.1647891Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:16:53.1648430Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:16:53.1648760Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:16:53.1649181Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:16:53.1649531Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:16:53.1649872Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-12-01T10:16:53.1650373Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:16:53.1650890Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:16:53.1651413Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:16:53.1651810Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:16:53.1652165Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:16:53.1652505Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:16:53.1652853Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:16:53.1653118Z dist init r=2, world=4
2022-12-01T10:16:53.1653293Z dist init r=0, world=4
2022-12-01T10:16:53.1653476Z dist init r=3, world=4
2022-12-01T10:16:53.1653659Z dist init r=1, world=4
2022-12-01T10:16:53.1653860Z skip: Need at least 2 CUDA devices (1.417s)
2022-12-01T10:16:53.1654296Z   test_grad_acc_configs_[(use_no_sync=True,num_iters=3),(use_no_sync=False,num_iters=3),(use_no_sync=True,num_iters=3)]_cpu_offload_CPUOffload(offload_params=True)_sharding_strategy_ShardingStrategy_NO_SHARD (__main__.TestGradAcc)
2022-12-01T10:16:53.1654797Z Tests gradient accumulation. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14925
2022-12-01T10:16:53.1655169Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14926
2022-12-01T10:16:53.1655491Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14927
2022-12-01T10:16:53.1655817Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 14928
2022-12-01T10:16:53.1656279Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:16:53.1656602Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:16:53.1657038Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:16:53.1657396Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:16:53.1657826Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:16:53.1658298Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:16:53.1658635Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:16:53.1659070Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:16:53.1659421Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:16:53.1659749Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:16:53.1660274Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:16:53.1660723Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:16:53.1661202Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:16:53.1661551Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:16:53.1661892Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:16:53.1662392Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:16:53.1662879Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:16:53.1663216Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:16:53.1663651Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:16:53.1664005Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:16:53.1664330Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-12-01T10:16:53.1664869Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:16:53.1665399Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:16:53.1665904Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:16:53.1666289Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:16:53.1666641Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:16:53.1666993Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:16:53.1667330Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:16:53.1667595Z dist init r=1, world=4
2022-12-01T10:16:53.1667783Z dist init r=2, world=4
2022-12-01T10:16:53.1667955Z dist init r=0, world=4
2022-12-01T10:16:53.1668139Z dist init r=3, world=4
2022-12-01T10:16:53.1668351Z skip: Need at least 2 CUDA devices (1.517s)
2022-12-01T10:16:53.1668792Z   test_grad_acc_configs_[(use_no_sync=True,num_iters=3),(use_no_sync=False,num_iters=3),(use_no_sync=True,num_iters=3)]_cpu_offload_CPUOffload(offload_params=True)_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestGradAcc)
2022-12-01T10:16:53.1669285Z Tests gradient accumulation. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14981
2022-12-01T10:16:53.1669653Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14982
2022-12-01T10:16:53.1669992Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14983
2022-12-01T10:16:53.1670309Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 14984
2022-12-01T10:16:53.1670770Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:16:53.1671106Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:16:53.1671541Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:16:53.1671881Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:16:53.1672222Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:16:53.1672695Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:16:53.1673027Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:16:53.1673489Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:16:53.1673876Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:16:53.1674218Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:16:53.1674673Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:16:53.1675007Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:16:53.1675438Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:16:53.1675789Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:16:53.1676114Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:16:53.1676619Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:16:53.1677125Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:16:53.1677458Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:16:53.1677876Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:16:53.1678227Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:16:53.1678568Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-12-01T10:16:53.1679054Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:16:53.1679584Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:16:53.1680111Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:16:53.1680507Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:16:53.1680847Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:16:53.1681192Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:16:53.1681541Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:16:53.1681790Z dist init r=0, world=4
2022-12-01T10:16:53.1681978Z dist init r=2, world=4
2022-12-01T10:16:53.1682163Z dist init r=3, world=4
2022-12-01T10:16:53.1682343Z dist init r=1, world=4
2022-12-01T10:16:53.1682542Z skip: Need at least 2 CUDA devices (1.418s)
2022-12-01T10:16:53.1682685Z 
2022-12-01T10:16:53.1682887Z ----------------------------------------------------------------------
2022-12-01T10:16:53.1683143Z Ran 12 tests in 17.446s
2022-12-01T10:16:53.1683250Z 
2022-12-01T10:16:53.1683348Z OK (skipped=12)
2022-12-01T10:16:53.1683461Z 
2022-12-01T10:16:53.1683549Z Generating XML reports...
2022-12-01T10:16:53.1683992Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_grad_acc/TEST-TestGradAcc-20221201101635.xml
2022-12-01T10:16:53.1684244Z 
2022-12-01T10:16:53.1684626Z ##[endgroup]
2022-12-01T10:16:53.1685094Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_grad_acc (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_grad_acc_aafxs4_0)
2022-12-01T10:16:53.1685360Z 
2022-12-01T10:16:53.1685555Z Running distributed/fsdp/test_fsdp_comm ... [2022-12-01 10:16:53.147163]
2022-12-01T10:16:53.1686066Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_comm.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-12-01 10:16:53.147545]
2022-12-01T10:17:06.1269170Z 
2022-12-01T10:17:06.1269952Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_comm
2022-12-01T10:17:06.1271324Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_comm (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_comm_71ucd83q)
2022-12-01T10:17:06.1272256Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_comm
2022-12-01T10:17:06.1272550Z 
2022-12-01T10:17:06.1272655Z Running tests...
2022-12-01T10:17:06.1273105Z ----------------------------------------------------------------------
2022-12-01T10:17:06.1273652Z   test_communication_nested_model_False_use_no_sync_False_sharding_strategy_None (__main__.TestCommunication)
2022-12-01T10:17:06.1274528Z Tests FSDP's communication cost in terms of calls to collective ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15048
2022-12-01T10:17:06.1275173Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15049
2022-12-01T10:17:06.1275781Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15050
2022-12-01T10:17:06.1276391Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 15051
2022-12-01T10:17:06.1277200Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:17:06.1277549Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:17:06.1277972Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:17:06.1278382Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:17:06.1278871Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:17:06.1279493Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:17:06.1279834Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:17:06.1280274Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:17:06.1280636Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:17:06.1280968Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:17:06.1281445Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:17:06.1281866Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:17:06.1282370Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:17:06.1282745Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:17:06.1283132Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:17:06.1283613Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:17:06.1283941Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:17:06.1284373Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:17:06.1284725Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:17:06.1285067Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-12-01T10:17:06.1285554Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:17:06.1286079Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:17:06.1286713Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:17:06.1287238Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:17:06.1287664Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:17:06.1288023Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:17:06.1288379Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:17:06.1288717Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:17:06.1288985Z dist init r=3, world=4
2022-12-01T10:17:06.1289177Z dist init r=2, world=4
2022-12-01T10:17:06.1289349Z dist init r=0, world=4
2022-12-01T10:17:06.1289533Z dist init r=1, world=4
2022-12-01T10:17:06.1289746Z skip: Need at least 2 CUDA devices (1.763s)
2022-12-01T10:17:06.1290111Z   test_communication_nested_model_False_use_no_sync_False_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestCommunication)
2022-12-01T10:17:06.1290668Z Tests FSDP's communication cost in terms of calls to collective ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15104
2022-12-01T10:17:06.1291070Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15105
2022-12-01T10:17:06.1291406Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15106
2022-12-01T10:17:06.1291727Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 15107
2022-12-01T10:17:06.1292188Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:17:06.1292530Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:17:06.1292968Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:17:06.1293311Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:17:06.1293659Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:17:06.1294137Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:17:06.1294470Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:17:06.1294887Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:17:06.1295237Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:17:06.1295579Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:17:06.1296034Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:17:06.1296364Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:17:06.1296798Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:17:06.1297152Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:17:06.1297556Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:17:06.1298035Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:17:06.1298492Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:17:06.1298925Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:17:06.1299265Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:17:06.1299609Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-12-01T10:17:06.1300155Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:17:06.1300912Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:17:06.1301439Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:17:06.1301961Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:17:06.1302355Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:17:06.1302699Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:17:06.1303050Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:17:06.1303400Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:17:06.1303666Z dist init r=0, world=4
2022-12-01T10:17:06.1303847Z dist init r=1, world=4
2022-12-01T10:17:06.1304033Z dist init r=2, world=4
2022-12-01T10:17:06.1304219Z dist init r=3, world=4
2022-12-01T10:17:06.1304417Z skip: Need at least 2 CUDA devices (1.418s)
2022-12-01T10:17:06.1304742Z   test_communication_nested_model_False_use_no_sync_True_sharding_strategy_None (__main__.TestCommunication)
2022-12-01T10:17:06.1305279Z Tests FSDP's communication cost in terms of calls to collective ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15160
2022-12-01T10:17:06.1305664Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15161
2022-12-01T10:17:06.1306002Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15162
2022-12-01T10:17:06.1306329Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 15163
2022-12-01T10:17:06.1306794Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:17:06.1307121Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:17:06.1307557Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:17:06.1307911Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:17:06.1308245Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:17:06.1308722Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:17:06.1309059Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:17:06.1309489Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:17:06.1309828Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:17:06.1310172Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:17:06.1310648Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:17:06.1310980Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:17:06.1311403Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:17:06.1311759Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:17:06.1312163Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-12-01T10:17:06.1312616Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:17:06.1312951Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:17:06.1313478Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:17:06.1313870Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:17:06.1314192Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:17:06.1314695Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:17:06.1315223Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:17:06.1315743Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:17:06.1316242Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:17:06.1316636Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:17:06.1316995Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:17:06.1317333Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:17:06.1317684Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:17:06.1317948Z dist init r=2, world=4
2022-12-01T10:17:06.1318139Z dist init r=3, world=4
2022-12-01T10:17:06.1318312Z dist init r=1, world=4
2022-12-01T10:17:06.1318498Z dist init r=0, world=4
2022-12-01T10:17:06.1318715Z skip: Need at least 2 CUDA devices (1.418s)
2022-12-01T10:17:06.1319059Z   test_communication_nested_model_False_use_no_sync_True_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestCommunication)
2022-12-01T10:17:06.1319623Z Tests FSDP's communication cost in terms of calls to collective ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15216
2022-12-01T10:17:06.1320024Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15217
2022-12-01T10:17:06.1320360Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15218
2022-12-01T10:17:06.1320680Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 15219
2022-12-01T10:17:06.1321136Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:17:06.1321480Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:17:06.1321901Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:17:06.1322254Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:17:06.1322638Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:17:06.1323116Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:17:06.1323442Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:17:06.1323876Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:17:06.1324227Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:17:06.1324554Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:17:06.1325030Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:17:06.1325366Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:17:06.1325797Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:17:06.1326176Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:17:06.1326525Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:17:06.1327062Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:17:06.1327568Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:17:06.1327886Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:17:06.1328322Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:17:06.1328678Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:17:06.1329008Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-12-01T10:17:06.1329514Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:17:06.1330047Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:17:06.1330570Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:17:06.1330948Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:17:06.1331305Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:17:06.1331655Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:17:06.1332008Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:17:06.1332259Z dist init r=0, world=4
2022-12-01T10:17:06.1332451Z dist init r=3, world=4
2022-12-01T10:17:06.1332638Z dist init r=2, world=4
2022-12-01T10:17:06.1332810Z dist init r=1, world=4
2022-12-01T10:17:06.1333027Z skip: Need at least 2 CUDA devices (1.418s)
2022-12-01T10:17:06.1333354Z   test_communication_nested_model_True_use_no_sync_False_sharding_strategy_None (__main__.TestCommunication)
2022-12-01T10:17:06.1333875Z Tests FSDP's communication cost in terms of calls to collective ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15272
2022-12-01T10:17:06.1334274Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15273
2022-12-01T10:17:06.1334609Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15274
2022-12-01T10:17:06.1334942Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 15275
2022-12-01T10:17:06.1335389Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:17:06.1335727Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:17:06.1336167Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:17:06.1336510Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:17:06.1336851Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:17:06.1337326Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:17:06.1337748Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:17:06.1338172Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:17:06.1338522Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:17:06.1338866Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:17:06.1339388Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:17:06.1339739Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:17:06.1340173Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:17:06.1340659Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:17:06.1340992Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:17:06.1341466Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:17:06.1341801Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:17:06.1342229Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:17:06.1342561Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:17:06.1342905Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-12-01T10:17:06.1343413Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:17:06.1343945Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:17:06.1344453Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:17:06.1344974Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:17:06.1345372Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:17:06.1345712Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:17:06.1346065Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:17:06.1346415Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:17:06.1346684Z dist init r=3, world=4
2022-12-01T10:17:06.1346860Z dist init r=0, world=4
2022-12-01T10:17:06.1347047Z dist init r=1, world=4
2022-12-01T10:17:06.1347233Z dist init r=2, world=4
2022-12-01T10:17:06.1347433Z skip: Need at least 2 CUDA devices (1.418s)
2022-12-01T10:17:06.1347788Z   test_communication_nested_model_True_use_no_sync_False_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestCommunication)
2022-12-01T10:17:06.1348355Z Tests FSDP's communication cost in terms of calls to collective ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15328
2022-12-01T10:17:06.1348762Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15329
2022-12-01T10:17:06.1349085Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15330
2022-12-01T10:17:06.1349413Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 15331
2022-12-01T10:17:06.1349878Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:17:06.1350202Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:17:06.1350641Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:17:06.1350995Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:17:06.1351340Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:17:06.1351802Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:17:06.1352136Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:17:06.1352643Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:17:06.1353048Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:17:06.1353389Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:17:06.1353866Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:17:06.1354201Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:17:06.1354620Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:17:06.1354973Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:17:06.1355320Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:17:06.1355797Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:17:06.1356118Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:17:06.1356550Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:17:06.1356903Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:17:06.1357231Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-12-01T10:17:06.1357735Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:17:06.1358259Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:17:06.1358783Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:17:06.1359295Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:17:06.1359692Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:17:06.1360048Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:17:06.1360399Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:17:06.1360734Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:17:06.1360996Z dist init r=3, world=4
2022-12-01T10:17:06.1361192Z dist init r=2, world=4
2022-12-01T10:17:06.1361361Z dist init r=1, world=4
2022-12-01T10:17:06.1361545Z dist init r=0, world=4
2022-12-01T10:17:06.1361758Z skip: Need at least 2 CUDA devices (1.418s)
2022-12-01T10:17:06.1362069Z   test_communication_nested_model_True_use_no_sync_True_sharding_strategy_None (__main__.TestCommunication)
2022-12-01T10:17:06.1362609Z Tests FSDP's communication cost in terms of calls to collective ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15384
2022-12-01T10:17:06.1363009Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15385
2022-12-01T10:17:06.1363345Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15386
2022-12-01T10:17:06.1363663Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 15387
2022-12-01T10:17:06.1364119Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:17:06.1364458Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:17:06.1364878Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:17:06.1365229Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:17:06.1365614Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:17:06.1366154Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:17:06.1366475Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:17:06.1366908Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:17:06.1367257Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:17:06.1367598Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:17:06.1368061Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:17:06.1368396Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:17:06.1368829Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:17:06.1369169Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:17:06.1369510Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:17:06.1369985Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:17:06.1370319Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:17:06.1370734Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:17:06.1371082Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:17:06.1371428Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-12-01T10:17:06.1371920Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:17:06.1372449Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:17:06.1372975Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:17:06.1373499Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:17:06.1373879Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:17:06.1374239Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:17:06.1374583Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:17:06.1374933Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:17:06.1375189Z dist init r=3, world=4
2022-12-01T10:17:06.1375381Z dist init r=2, world=4
2022-12-01T10:17:06.1375570Z dist init r=1, world=4
2022-12-01T10:17:06.1375742Z dist init r=0, world=4
2022-12-01T10:17:06.1375956Z skip: Need at least 2 CUDA devices (1.418s)
2022-12-01T10:17:06.1376310Z   test_communication_nested_model_True_use_no_sync_True_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestCommunication)
2022-12-01T10:17:06.1376871Z Tests FSDP's communication cost in terms of calls to collective ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15440
2022-12-01T10:17:06.1377255Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15441
2022-12-01T10:17:06.1377679Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15442
2022-12-01T10:17:06.1378009Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 15443
2022-12-01T10:17:06.1378511Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:17:06.1378853Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:17:06.1379329Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:17:06.1379685Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:17:06.1380013Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:17:06.1380632Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:17:06.1380972Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:17:06.1381395Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:17:06.1381746Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:17:06.1382094Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:17:06.1382568Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:17:06.1382888Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:17:06.1383322Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:17:06.1383672Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:17:06.1384012Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-12-01T10:17:06.1384466Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:17:06.1384802Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:17:06.1385233Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:17:06.1385569Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:17:06.1385911Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:17:06.1386414Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:17:06.1386933Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:17:06.1387445Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:17:06.1387967Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:17:06.1388365Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:17:06.1388722Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:17:06.1389062Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:17:06.1389409Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:17:06.1389679Z dist init r=1, world=4
2022-12-01T10:17:06.1389854Z dist init r=0, world=4
2022-12-01T10:17:06.1390040Z dist init r=2, world=4
2022-12-01T10:17:06.1390223Z dist init r=3, world=4
2022-12-01T10:17:06.1390422Z skip: Need at least 2 CUDA devices (1.418s)
2022-12-01T10:17:06.1390563Z 
2022-12-01T10:17:06.1390764Z ----------------------------------------------------------------------
2022-12-01T10:17:06.1391020Z Ran 8 tests in 11.690s
2022-12-01T10:17:06.1391137Z 
2022-12-01T10:17:06.1391213Z OK (skipped=8)
2022-12-01T10:17:06.1391310Z 
2022-12-01T10:17:06.1391399Z Generating XML reports...
2022-12-01T10:17:06.1391921Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_comm/TEST-TestCommunication-20221201101654.xml
2022-12-01T10:17:06.1392223Z 
2022-12-01T10:17:06.1392569Z ##[endgroup]
2022-12-01T10:17:06.1393006Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_comm (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_comm_71ucd83q)
2022-12-01T10:17:06.1393261Z 
2022-12-01T10:17:06.1393482Z Running distributed/fsdp/test_fsdp_sharded_grad_scaler ... [2022-12-01 10:17:06.127776]
2022-12-01T10:17:06.1394039Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_sharded_grad_scaler.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-12-01 10:17:06.128113]
2022-12-01T10:17:18.8916182Z 
2022-12-01T10:17:18.8917063Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_sharded_grad_scaler
2022-12-01T10:17:18.8918216Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_sharded_grad_scaler (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_sharded_grad_scaler_22bs1els)
2022-12-01T10:17:18.8919272Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_sharded_grad_scaler
2022-12-01T10:17:18.8919883Z 
2022-12-01T10:17:18.8920019Z Running tests...
2022-12-01T10:17:18.8920567Z ----------------------------------------------------------------------
2022-12-01T10:17:18.8921146Z   test_grad_scaling (__main__.TestShardGradScaler) ... skip: no supported device (cuda, xla) found (0.001s)
2022-12-01T10:17:18.8921853Z   test_inf_gradients_skip_optim_step (__main__.TestShardGradScaler) ... skip: no supported device (cuda, xla) found (0.001s)
2022-12-01T10:17:18.8922578Z   test_scaling_unscaling_sparse (__main__.TestShardGradScaler) ... skip: no supported device (cuda, xla) found (0.002s)
2022-12-01T10:17:18.8923478Z   test_fsdp_ddp_parity_with_grad_scaler_offload_false_none_mixed_precision (__main__.TestShardedGradScalerParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15507
2022-12-01T10:17:18.8924186Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15508
2022-12-01T10:17:18.8924753Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15509
2022-12-01T10:17:18.8925304Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 15510
2022-12-01T10:17:18.8926080Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:17:18.8926642Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:17:18.8927323Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:17:18.8927882Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:17:18.8928396Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:17:18.8929203Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:17:18.8929704Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:17:18.8930309Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:17:18.8930762Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:17:18.8931229Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:17:18.8931899Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:17:18.8932344Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:17:18.8932957Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:17:18.8933682Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:17:18.8934160Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:17:18.8934941Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:17:18.8935461Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:17:18.8936134Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:17:18.8936625Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:17:18.8937123Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-12-01T10:17:18.8937974Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:17:18.8938793Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:17:18.8939638Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:17:18.8940675Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:17:18.8941281Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:17:18.8941840Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:17:18.8942346Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:17:18.8942874Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:17:18.8943271Z dist init r=2, world=4
2022-12-01T10:17:18.8943545Z dist init r=1, world=4
2022-12-01T10:17:18.8943831Z dist init r=3, world=4
2022-12-01T10:17:18.8944133Z dist init r=0, world=4
2022-12-01T10:17:18.8944461Z skip: Need at least 2 CUDA devices (1.654s)
2022-12-01T10:17:18.8945115Z   test_fsdp_ddp_parity_with_grad_scaler_offload_false_none_none (__main__.TestShardedGradScalerParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15563
2022-12-01T10:17:18.8945783Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15564
2022-12-01T10:17:18.8946251Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15565
2022-12-01T10:17:18.8946726Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 15566
2022-12-01T10:17:18.8947424Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:17:18.8947905Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:17:18.8948501Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:17:18.8949001Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:17:18.8949529Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:17:18.8950215Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:17:18.8950669Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:17:18.8951331Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:17:18.8951906Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:17:18.8952451Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:17:18.8953018Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:17:18.8953500Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:17:18.8953994Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:17:18.8954349Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:17:18.8954678Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:17:18.8955156Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:17:18.8955489Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:17:18.8955910Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:17:18.8956267Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:17:18.8956611Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-12-01T10:17:18.8957117Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:17:18.8957789Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:17:18.8958330Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:17:18.8958855Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:17:18.8959247Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:17:18.8959587Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:17:18.8959930Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:17:18.8960290Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:17:18.8960545Z dist init r=1, world=4
2022-12-01T10:17:18.8960731Z dist init r=2, world=4
2022-12-01T10:17:18.8960914Z dist init r=3, world=4
2022-12-01T10:17:18.8961083Z dist init r=0, world=4
2022-12-01T10:17:18.8961296Z skip: Need at least 2 CUDA devices (1.318s)
2022-12-01T10:17:18.8961756Z   test_fsdp_ddp_parity_with_grad_scaler_offload_false_shard_grad_op_mixed_precision (__main__.TestShardedGradScalerParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15619
2022-12-01T10:17:18.8962242Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15620
2022-12-01T10:17:18.8962562Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15621
2022-12-01T10:17:18.8962887Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 15622
2022-12-01T10:17:18.8963348Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:17:18.8963677Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:17:18.8964112Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:17:18.8964467Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:17:18.8964810Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:17:18.8965268Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:17:18.8965600Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:17:18.8966032Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:17:18.8966433Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:17:18.8966763Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:17:18.8967273Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:17:18.8967598Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:17:18.8968011Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:17:18.8968357Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:17:18.8968690Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-12-01T10:17:18.8969155Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:17:18.8969468Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:17:18.8969897Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:17:18.8970247Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:17:18.8970567Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:17:18.8971060Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:17:18.8971863Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:17:18.8972439Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:17:18.8972937Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:17:18.8973326Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:17:18.8973685Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:17:18.8974034Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:17:18.8974368Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:17:18.8974630Z dist init r=3, world=4
2022-12-01T10:17:18.8974816Z dist init r=1, world=4
2022-12-01T10:17:18.8974982Z dist init r=2, world=4
2022-12-01T10:17:18.8975168Z dist init r=0, world=4
2022-12-01T10:17:18.8975377Z skip: Need at least 2 CUDA devices (1.417s)
2022-12-01T10:17:18.8975803Z   test_fsdp_ddp_parity_with_grad_scaler_offload_false_shard_grad_op_none (__main__.TestShardedGradScalerParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15675
2022-12-01T10:17:18.8976286Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15676
2022-12-01T10:17:18.8976619Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15677
2022-12-01T10:17:18.8976950Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 15678
2022-12-01T10:17:18.8977389Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:17:18.8977838Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:17:18.8978273Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:17:18.8978621Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:17:18.8978945Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:17:18.8979416Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:17:18.8979818Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:17:18.8980277Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:17:18.8980789Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:17:18.8981134Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:17:18.8981608Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:17:18.8981923Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:17:18.8982353Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:17:18.8982705Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:17:18.8983043Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:17:18.8983493Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:17:18.8983823Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:17:18.8984245Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:17:18.8984574Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:17:18.8984907Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-12-01T10:17:18.8985399Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:17:18.8985921Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:17:18.8986428Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:17:18.8986941Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:17:18.8987330Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:17:18.8987672Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:17:18.8997904Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:17:18.8998277Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:17:18.8998536Z dist init r=1, world=4
2022-12-01T10:17:18.8998727Z dist init r=0, world=4
2022-12-01T10:17:18.8998911Z dist init r=2, world=4
2022-12-01T10:17:18.8999082Z dist init r=3, world=4
2022-12-01T10:17:18.8999297Z skip: Need at least 2 CUDA devices (1.417s)
2022-12-01T10:17:18.8999749Z   test_fsdp_ddp_parity_with_grad_scaler_offload_true_none_mixed_precision (__main__.TestShardedGradScalerParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15731
2022-12-01T10:17:18.9000226Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15732
2022-12-01T10:17:18.9000548Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15733
2022-12-01T10:17:18.9000877Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 15734
2022-12-01T10:17:18.9001374Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:17:18.9001698Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:17:18.9002132Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:17:18.9002484Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:17:18.9003010Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:17:18.9003529Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:17:18.9003859Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:17:18.9004290Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:17:18.9004625Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:17:18.9004964Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:17:18.9005434Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:17:18.9005763Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:17:18.9006182Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:17:18.9006536Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:17:18.9006870Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-12-01T10:17:18.9007367Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:17:18.9007876Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:17:18.9008370Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:17:18.9008702Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:17:18.9009117Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:17:18.9009464Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:17:18.9009802Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:17:18.9010298Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:17:18.9010806Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:17:18.9011191Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:17:18.9011539Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:17:18.9011883Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:17:18.9012217Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:17:18.9012474Z dist init r=2, world=4
2022-12-01T10:17:18.9012662Z dist init r=1, world=4
2022-12-01T10:17:18.9012832Z dist init r=3, world=4
2022-12-01T10:17:18.9013010Z dist init r=0, world=4
2022-12-01T10:17:18.9013219Z skip: Need at least 2 CUDA devices (1.417s)
2022-12-01T10:17:18.9013631Z   test_fsdp_ddp_parity_with_grad_scaler_offload_true_none_none (__main__.TestShardedGradScalerParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15787
2022-12-01T10:17:18.9014088Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15788
2022-12-01T10:17:18.9014419Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15789
2022-12-01T10:17:18.9014751Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 15790
2022-12-01T10:17:18.9015188Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:17:18.9015556Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:17:18.9016018Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:17:18.9016366Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:17:18.9016696Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:17:18.9017164Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:17:18.9017582Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:17:18.9018006Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:17:18.9018354Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:17:18.9018698Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:17:18.9019164Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:17:18.9019481Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:17:18.9019908Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:17:18.9020255Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:17:18.9020746Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:17:18.9021221Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:17:18.9021550Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:17:18.9021978Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:17:18.9022312Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:17:18.9022653Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-12-01T10:17:18.9023150Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:17:18.9023667Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:17:18.9024168Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:17:18.9024686Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:17:18.9025072Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:17:18.9025411Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:17:18.9025762Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:17:18.9026106Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:17:18.9026366Z dist init r=0, world=4
2022-12-01T10:17:18.9026538Z dist init r=3, world=4
2022-12-01T10:17:18.9026716Z dist init r=2, world=4
2022-12-01T10:17:18.9026895Z dist init r=1, world=4
2022-12-01T10:17:18.9027091Z skip: Need at least 2 CUDA devices (1.417s)
2022-12-01T10:17:18.9027537Z   test_fsdp_ddp_parity_with_grad_scaler_offload_true_shard_grad_op_mixed_precision (__main__.TestShardedGradScalerParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15843
2022-12-01T10:17:18.9028010Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15844
2022-12-01T10:17:18.9028341Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15845
2022-12-01T10:17:18.9028729Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 15846
2022-12-01T10:17:18.9029240Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:17:18.9029576Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:17:18.9029992Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:17:18.9030339Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:17:18.9030681Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:17:18.9031149Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:17:18.9031462Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:17:18.9031891Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:17:18.9032240Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:17:18.9032562Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:17:18.9033034Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:17:18.9033363Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:17:18.9033788Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:17:18.9034123Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:17:18.9034460Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-12-01T10:17:18.9034928Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:17:18.9035259Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:17:18.9035674Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:17:18.9036019Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:17:18.9036352Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:17:18.9036832Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:17:18.9037349Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:17:18.9037860Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:17:18.9038377Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:17:18.9038753Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:17:18.9039099Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:17:18.9039440Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:17:18.9039788Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:17:18.9040033Z dist init r=2, world=4
2022-12-01T10:17:18.9040217Z dist init r=0, world=4
2022-12-01T10:17:18.9040396Z dist init r=1, world=4
2022-12-01T10:17:18.9040562Z dist init r=3, world=4
2022-12-01T10:17:18.9040773Z skip: Need at least 2 CUDA devices (1.417s)
2022-12-01T10:17:18.9041251Z   test_fsdp_ddp_parity_with_grad_scaler_offload_true_shard_grad_op_none (__main__.TestShardedGradScalerParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15899
2022-12-01T10:17:18.9041736Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15900
2022-12-01T10:17:18.9042069Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15901
2022-12-01T10:17:18.9042394Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 15902
2022-12-01T10:17:18.9042848Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:17:18.9043166Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:17:18.9043594Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:17:18.9043942Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:17:18.9044282Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:17:18.9044739Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:17:18.9045068Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:17:18.9045490Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:17:18.9045819Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:17:18.9046158Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:17:18.9046622Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:17:18.9046946Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:17:18.9047355Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:17:18.9047704Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:17:18.9048044Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-12-01T10:17:18.9048586Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:17:18.9049066Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:17:18.9049392Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:17:18.9049821Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:17:18.9050167Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:17:18.9050491Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:17:18.9050988Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:17:18.9051512Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:17:18.9052028Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:17:18.9052401Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:17:18.9052752Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:17:18.9053097Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:17:18.9053430Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:17:18.9053690Z dist init r=3, world=4
2022-12-01T10:17:18.9053874Z dist init r=2, world=4
2022-12-01T10:17:18.9054082Z dist init r=1, world=4
2022-12-01T10:17:18.9054263Z dist init r=0, world=4
2022-12-01T10:17:18.9054504Z skip: Need at least 2 CUDA devices (1.417s)
2022-12-01T10:17:18.9054642Z 
2022-12-01T10:17:18.9054843Z ----------------------------------------------------------------------
2022-12-01T10:17:18.9055079Z Ran 11 tests in 11.478s
2022-12-01T10:17:18.9055193Z 
2022-12-01T10:17:18.9055267Z OK (skipped=11)
2022-12-01T10:17:18.9055376Z 
2022-12-01T10:17:18.9055462Z Generating XML reports...
2022-12-01T10:17:18.9055934Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_sharded_grad_scaler/TEST-TestShardGradScaler-20221201101707.xml
2022-12-01T10:17:18.9056618Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_sharded_grad_scaler/TEST-TestShardedGradScalerParityWithDDP-20221201101707.xml
2022-12-01T10:17:18.9056934Z 
2022-12-01T10:17:18.9058826Z ##[endgroup]
2022-12-01T10:17:18.9059339Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_sharded_grad_scaler (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_sharded_grad_scaler_22bs1els)
2022-12-01T10:17:18.9059636Z 
2022-12-01T10:17:18.9059836Z Running distributed/test_c10d_common ... [2022-12-01 10:17:18.892350]
2022-12-01T10:17:18.9060576Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_c10d_common.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-12-01 10:17:18.892756]
2022-12-01T10:17:47.8988969Z 
2022-12-01T10:17:47.8989554Z Expand the folded group to see the log file of distributed/test_c10d_common
2022-12-01T10:17:47.8990897Z ##[group]PRINTING LOG FILE of distributed/test_c10d_common (/var/lib/jenkins/workspace/test/test-reports/distributed-test_c10d_common_jynjicd9)
2022-12-01T10:17:47.8991492Z <unittest.suite.TestSuite tests=[<__main__.CommTest testMethod=test_debug_level>]>
2022-12-01T10:17:47.8991931Z test_debug_level (__main__.CommTest)
2022-12-01T10:17:47.8992891Z <unittest.suite.TestSuite tests=[<__main__.ComputeBucketAssignmentTest testMethod=test_multi_limit_multi_dtype>, <__main__.ComputeBucketAssignmentTest testMethod=test_multi_limit_single_dtype>, <__main__.ComputeBucketAssignmentTest testMethod=test_single_limit_multi_dtype>, <__main__.ComputeBucketAssignmentTest testMethod=test_single_limit_single_dtype>]>
2022-12-01T10:17:47.8994010Z test_multi_limit_multi_dtype (__main__.ComputeBucketAssignmentTest)
2022-12-01T10:17:47.8994580Z test_multi_limit_single_dtype (__main__.ComputeBucketAssignmentTest)
2022-12-01T10:17:47.8995144Z test_single_limit_multi_dtype (__main__.ComputeBucketAssignmentTest)
2022-12-01T10:17:47.8995686Z test_single_limit_single_dtype (__main__.ComputeBucketAssignmentTest)
2022-12-01T10:17:47.8996842Z <unittest.suite.TestSuite tests=[<__main__.PythonProcessGroupExtensionTest testMethod=test_backend_class_attr>, <__main__.PythonProcessGroupExtensionTest testMethod=test_collectives>, <__main__.PythonProcessGroupExtensionTest testMethod=test_get_backend_name>, <__main__.PythonProcessGroupExtensionTest testMethod=test_send_recv>]>
2022-12-01T10:17:47.8998034Z test_backend_class_attr (__main__.PythonProcessGroupExtensionTest)
2022-12-01T10:17:47.8998576Z test_collectives (__main__.PythonProcessGroupExtensionTest)
2022-12-01T10:17:47.8999051Z test_get_backend_name (__main__.PythonProcessGroupExtensionTest)
2022-12-01T10:17:47.8999561Z test_send_recv (__main__.PythonProcessGroupExtensionTest)
2022-12-01T10:17:47.9000139Z <unittest.suite.TestSuite tests=[<__main__.ReduceOpTest testMethod=test_op_isinstance_of_reduceop>]>
2022-12-01T10:17:47.9000636Z test_op_isinstance_of_reduceop (__main__.ReduceOpTest)
2022-12-01T10:17:47.9001358Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_common
2022-12-01T10:17:47.9002127Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:17:47.9002665Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:17:47.9003490Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:17:47.9004021Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:17:47.9004248Z 
2022-12-01T10:17:47.9004352Z Running tests...
2022-12-01T10:17:47.9004781Z ----------------------------------------------------------------------
2022-12-01T10:17:47.9005285Z   test_debug_level (__main__.CommTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15976
2022-12-01T10:17:47.9005805Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15977
2022-12-01T10:17:47.9006450Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:17:47.9006909Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:17:47.9007505Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:17:47.9007993Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:17:47.9008476Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:17:47.9009189Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:17:47.9009689Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:17:47.9010327Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:17:47.9010823Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:17:47.9011257Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:17:47.9011648Z ok (1.760s)
2022-12-01T10:17:47.9011808Z 
2022-12-01T10:17:47.9012114Z ----------------------------------------------------------------------
2022-12-01T10:17:47.9012448Z Ran 1 test in 1.760s
2022-12-01T10:17:47.9012634Z 
2022-12-01T10:17:47.9012744Z OK
2022-12-01T10:17:47.9012900Z 
2022-12-01T10:17:47.9013041Z Generating XML reports...
2022-12-01T10:17:47.9013719Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_common/TEST-CommTest-20221201101721.xml
2022-12-01T10:17:47.9014401Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_common
2022-12-01T10:17:47.9015155Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:17:47.9015649Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:17:47.9016283Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:17:47.9016854Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:17:47.9017136Z 
2022-12-01T10:17:47.9017259Z Running tests...
2022-12-01T10:17:47.9017847Z ----------------------------------------------------------------------
2022-12-01T10:17:47.9018325Z   test_multi_limit_multi_dtype (__main__.ComputeBucketAssignmentTest) ... ok (0.243s)
2022-12-01T10:17:47.9018595Z 
2022-12-01T10:17:47.9018882Z ----------------------------------------------------------------------
2022-12-01T10:17:47.9019203Z Ran 1 test in 0.244s
2022-12-01T10:17:47.9019361Z 
2022-12-01T10:17:47.9019431Z OK
2022-12-01T10:17:47.9019557Z 
2022-12-01T10:17:47.9019680Z Generating XML reports...
2022-12-01T10:17:47.9020568Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_common/TEST-ComputeBucketAssignmentTest-20221201101724.xml
2022-12-01T10:17:47.9021278Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_common
2022-12-01T10:17:47.9021900Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:17:47.9022370Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:17:47.9023200Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:17:47.9023776Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:17:47.9024004Z 
2022-12-01T10:17:47.9024110Z Running tests...
2022-12-01T10:17:47.9024560Z ----------------------------------------------------------------------
2022-12-01T10:17:47.9025064Z   test_multi_limit_single_dtype (__main__.ComputeBucketAssignmentTest) ... ok (0.247s)
2022-12-01T10:17:47.9025388Z 
2022-12-01T10:17:47.9025702Z ----------------------------------------------------------------------
2022-12-01T10:17:47.9026098Z Ran 1 test in 0.247s
2022-12-01T10:17:47.9026276Z 
2022-12-01T10:17:47.9026348Z OK
2022-12-01T10:17:47.9026440Z 
2022-12-01T10:17:47.9026526Z Generating XML reports...
2022-12-01T10:17:47.9026992Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_common/TEST-ComputeBucketAssignmentTest-20221201101726.xml
2022-12-01T10:17:47.9027502Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_common
2022-12-01T10:17:47.9027966Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:17:47.9028284Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:17:47.9028708Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:17:47.9029053Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:17:47.9029218Z 
2022-12-01T10:17:47.9029291Z Running tests...
2022-12-01T10:17:47.9029583Z ----------------------------------------------------------------------
2022-12-01T10:17:47.9029913Z   test_single_limit_multi_dtype (__main__.ComputeBucketAssignmentTest) ... ok (0.249s)
2022-12-01T10:17:47.9030108Z 
2022-12-01T10:17:47.9030308Z ----------------------------------------------------------------------
2022-12-01T10:17:47.9030538Z Ran 1 test in 0.250s
2022-12-01T10:17:47.9030652Z 
2022-12-01T10:17:47.9030714Z OK
2022-12-01T10:17:47.9030807Z 
2022-12-01T10:17:47.9030891Z Generating XML reports...
2022-12-01T10:17:47.9031359Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_common/TEST-ComputeBucketAssignmentTest-20221201101728.xml
2022-12-01T10:17:47.9031855Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_common
2022-12-01T10:17:47.9032316Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:17:47.9032649Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:17:47.9033065Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:17:47.9033407Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:17:47.9033574Z 
2022-12-01T10:17:47.9033649Z Running tests...
2022-12-01T10:17:47.9033950Z ----------------------------------------------------------------------
2022-12-01T10:17:47.9034269Z   test_single_limit_single_dtype (__main__.ComputeBucketAssignmentTest) ... ok (0.248s)
2022-12-01T10:17:47.9034460Z 
2022-12-01T10:17:47.9034656Z ----------------------------------------------------------------------
2022-12-01T10:17:47.9034893Z Ran 1 test in 0.248s
2022-12-01T10:17:47.9035003Z 
2022-12-01T10:17:47.9035049Z OK
2022-12-01T10:17:47.9035142Z 
2022-12-01T10:17:47.9035224Z Generating XML reports...
2022-12-01T10:17:47.9035690Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_common/TEST-ComputeBucketAssignmentTest-20221201101730.xml
2022-12-01T10:17:47.9036186Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_common
2022-12-01T10:17:47.9036637Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:17:47.9037045Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:17:47.9037523Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:17:47.9037877Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:17:47.9038027Z 
2022-12-01T10:17:47.9038102Z Running tests...
2022-12-01T10:17:47.9038404Z ----------------------------------------------------------------------
2022-12-01T10:17:47.9038824Z   test_backend_class_attr (__main__.PythonProcessGroupExtensionTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16049
2022-12-01T10:17:47.9039228Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16050
2022-12-01T10:17:47.9039563Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16051
2022-12-01T10:17:47.9039896Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 16052
2022-12-01T10:17:47.9040354Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:17:47.9040673Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:17:47.9041100Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:17:47.9041445Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:17:47.9041754Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:17:47.9042212Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:17:47.9042540Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:17:47.9042968Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:17:47.9043300Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:17:47.9043621Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:17:47.9044113Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:17:47.9044650Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:17:47.9045162Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:17:47.9045507Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:17:47.9045829Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:17:47.9046279Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:17:47.9046607Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:17:47.9047035Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:17:47.9047381Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:17:47.9047687Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:17:47.9047938Z ok (1.967s)
2022-12-01T10:17:47.9048042Z 
2022-12-01T10:17:47.9048243Z ----------------------------------------------------------------------
2022-12-01T10:17:47.9048470Z Ran 1 test in 1.967s
2022-12-01T10:17:47.9048581Z 
2022-12-01T10:17:47.9048643Z OK
2022-12-01T10:17:47.9048734Z 
2022-12-01T10:17:47.9048820Z Generating XML reports...
2022-12-01T10:17:47.9049293Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_common/TEST-PythonProcessGroupExtensionTest-20221201101731.xml
2022-12-01T10:17:47.9049814Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_common
2022-12-01T10:17:47.9050329Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:17:47.9050697Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:17:47.9051116Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:17:47.9051464Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:17:47.9051630Z 
2022-12-01T10:17:47.9051708Z Running tests...
2022-12-01T10:17:47.9052012Z ----------------------------------------------------------------------
2022-12-01T10:17:47.9052414Z   test_collectives (__main__.PythonProcessGroupExtensionTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16104
2022-12-01T10:17:47.9052829Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16105
2022-12-01T10:17:47.9053168Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16106
2022-12-01T10:17:47.9053487Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 16107
2022-12-01T10:17:47.9053942Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:17:47.9054273Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:17:47.9054699Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:17:47.9055032Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:17:47.9055351Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:17:47.9055816Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:17:47.9056143Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:17:47.9056562Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:17:47.9056910Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:17:47.9057231Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:17:47.9057676Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-12-01T10:17:47.9058151Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:17:47.9058483Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:17:47.9058918Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:17:47.9059249Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:17:47.9059577Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:17:47.9059941Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:17:47.9060541Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:17:47.9060885Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:17:47.9061321Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:17:47.9061671Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:17:47.9061981Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:17:47.9062346Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:17:47.9062719Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:17:47.9063294Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:17:47.9063857Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:17:47.9064376Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:17:47.9064898Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:17:47.9065172Z ok (1.965s)
2022-12-01T10:17:47.9065278Z 
2022-12-01T10:17:47.9065479Z ----------------------------------------------------------------------
2022-12-01T10:17:47.9065724Z Ran 1 test in 1.966s
2022-12-01T10:17:47.9065840Z 
2022-12-01T10:17:47.9065902Z OK
2022-12-01T10:17:47.9065980Z 
2022-12-01T10:17:47.9066066Z Generating XML reports...
2022-12-01T10:17:47.9066553Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_common/TEST-PythonProcessGroupExtensionTest-20221201101735.xml
2022-12-01T10:17:47.9067075Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_common
2022-12-01T10:17:47.9067521Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:17:47.9067856Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:17:47.9068286Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:17:47.9068630Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:17:47.9068794Z 
2022-12-01T10:17:47.9068854Z Running tests...
2022-12-01T10:17:47.9069153Z ----------------------------------------------------------------------
2022-12-01T10:17:47.9069569Z   test_get_backend_name (__main__.PythonProcessGroupExtensionTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16164
2022-12-01T10:17:47.9069982Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16165
2022-12-01T10:17:47.9070300Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16166
2022-12-01T10:17:47.9070618Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 16167
2022-12-01T10:17:47.9071068Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:17:47.9071385Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:17:47.9071811Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:17:47.9072156Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:17:47.9072476Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:17:47.9072923Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:17:47.9073254Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:17:47.9073682Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:17:47.9074013Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:17:47.9074336Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:17:47.9074795Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:17:47.9075120Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:17:47.9075534Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:17:47.9075916Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:17:47.9076245Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:17:47.9076733Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:17:47.9077046Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:17:47.9077470Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:17:47.9077815Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:17:47.9078127Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:17:47.9078376Z ok (1.965s)
2022-12-01T10:17:47.9078480Z 
2022-12-01T10:17:47.9078684Z ----------------------------------------------------------------------
2022-12-01T10:17:47.9078926Z Ran 1 test in 1.965s
2022-12-01T10:17:47.9079025Z 
2022-12-01T10:17:47.9079088Z OK
2022-12-01T10:17:47.9079185Z 
2022-12-01T10:17:47.9079273Z Generating XML reports...
2022-12-01T10:17:47.9079764Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_common/TEST-PythonProcessGroupExtensionTest-20221201101739.xml
2022-12-01T10:17:47.9080268Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_common
2022-12-01T10:17:47.9080726Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:17:47.9081054Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:17:47.9081476Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:17:47.9081809Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:17:47.9081974Z 
2022-12-01T10:17:47.9082045Z Running tests...
2022-12-01T10:17:47.9082350Z ----------------------------------------------------------------------
2022-12-01T10:17:47.9082745Z   test_send_recv (__main__.PythonProcessGroupExtensionTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16219
2022-12-01T10:17:47.9083156Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16220
2022-12-01T10:17:47.9083485Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16221
2022-12-01T10:17:47.9083807Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 16222
2022-12-01T10:17:47.9084243Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:17:47.9084569Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:17:47.9084992Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:17:47.9085323Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:17:47.9085646Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:17:47.9086126Z [W socket.cpp:601] [c10d] The client socket has failed to connect to [localhost]:6789 (errno: 99 - Cannot assign requested address).
2022-12-01T10:17:47.9086603Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:17:47.9086913Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:17:47.9087336Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:17:47.9087684Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:17:47.9088007Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:17:47.9088451Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:17:47.9088820Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:17:47.9089302Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:17:47.9089631Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:17:47.9089955Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:17:47.9090322Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-12-01T10:17:47.9090794Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:17:47.9091110Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:17:47.9091540Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:17:47.9091888Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:17:47.9092199Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:17:47.9092570Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:17:47.9092935Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:17:47.9093297Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:17:47.9093774Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:17:47.9094299Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:17:47.9094814Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:17:47.9095330Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:17:47.9095610Z ok (2.871s)
2022-12-01T10:17:47.9095713Z 
2022-12-01T10:17:47.9095910Z ----------------------------------------------------------------------
2022-12-01T10:17:47.9096149Z Ran 1 test in 2.871s
2022-12-01T10:17:47.9096261Z 
2022-12-01T10:17:47.9096308Z OK
2022-12-01T10:17:47.9096399Z 
2022-12-01T10:17:47.9096483Z Generating XML reports...
2022-12-01T10:17:47.9096968Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_common/TEST-PythonProcessGroupExtensionTest-20221201101742.xml
2022-12-01T10:17:47.9097577Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_common
2022-12-01T10:17:47.9098036Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:17:47.9098368Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:17:47.9098800Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:17:47.9099152Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:17:47.9099303Z 
2022-12-01T10:17:47.9099379Z Running tests...
2022-12-01T10:17:47.9099684Z ----------------------------------------------------------------------
2022-12-01T10:17:47.9099988Z   test_op_isinstance_of_reduceop (__main__.ReduceOpTest) ... ok (0.246s)
2022-12-01T10:17:47.9100153Z 
2022-12-01T10:17:47.9100332Z ----------------------------------------------------------------------
2022-12-01T10:17:47.9100735Z Ran 1 test in 0.247s
2022-12-01T10:17:47.9100852Z 
2022-12-01T10:17:47.9100914Z OK
2022-12-01T10:17:47.9101008Z 
2022-12-01T10:17:47.9101093Z Generating XML reports...
2022-12-01T10:17:47.9101502Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_common/TEST-ReduceOpTest-20221201101747.xml
2022-12-01T10:17:47.9101746Z 
2022-12-01T10:17:47.9102178Z ##[endgroup]
2022-12-01T10:17:47.9102645Z FINISHED PRINTING LOG FILE of distributed/test_c10d_common (/var/lib/jenkins/workspace/test/test-reports/distributed-test_c10d_common_jynjicd9)
2022-12-01T10:17:47.9102889Z 
2022-12-01T10:17:47.9103130Z Running distributed/fsdp/test_fsdp_tp_integration ... [2022-12-01 10:17:47.899687]
2022-12-01T10:17:47.9103670Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_tp_integration.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-12-01 10:17:47.900080]
2022-12-01T10:17:56.6237015Z 
2022-12-01T10:17:56.6238784Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_tp_integration
2022-12-01T10:17:56.6240098Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_tp_integration (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_tp_integration_gyh31wac)
2022-12-01T10:17:56.6241089Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_tp_integration
2022-12-01T10:17:56.6241402Z 
2022-12-01T10:17:56.6241501Z Running tests...
2022-12-01T10:17:56.6242131Z ----------------------------------------------------------------------
2022-12-01T10:17:56.6242598Z   test_fsdp_tp_checkpoint_integration (__main__.TestTPFSDPIntegration)
2022-12-01T10:17:56.6243164Z Tests checkpointing for TP + FSDP integration. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16289
2022-12-01T10:17:56.6243740Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16290
2022-12-01T10:17:56.6244292Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16291
2022-12-01T10:17:56.6245159Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 16292
2022-12-01T10:17:56.6245941Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:17:56.6246455Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:17:56.6247202Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:17:56.6247781Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:17:56.6248349Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:17:56.6249165Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:17:56.6249760Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:17:56.6250587Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:17:56.6251196Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:17:56.6251747Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:17:56.6252601Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:17:56.6253213Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:17:56.6253988Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:17:56.6254596Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:17:56.6255184Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-12-01T10:17:56.6256034Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:17:56.6256589Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:17:56.6266766Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:17:56.6267653Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:17:56.6268383Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:17:56.6269127Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:17:56.6270019Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:17:56.6270963Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:17:56.6271921Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:17:56.6272351Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:17:56.6272939Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:17:56.6274141Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:17:56.6274485Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:17:56.6274745Z dist init r=3, world=4
2022-12-01T10:17:56.6274936Z dist init r=2, world=4
2022-12-01T10:17:56.6275112Z dist init r=0, world=4
2022-12-01T10:17:56.6275287Z dist init r=1, world=4
2022-12-01T10:17:56.6275482Z skip: Need at least 4 CUDA devices (1.655s)
2022-12-01T10:17:56.6275856Z   test_fsdp_tp_integration_tensor_parallel_size_2_cpu_offload_CPUOffload(offload_params=False) (__main__.TestTPFSDPIntegration)
2022-12-01T10:17:56.6276952Z Tests training for TP + FSDP integration by comparing an FSDP-only ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16345
2022-12-01T10:17:56.6277412Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16346
2022-12-01T10:17:56.6278648Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16347
2022-12-01T10:17:56.6278977Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 16348
2022-12-01T10:17:56.6279686Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:17:56.6280281Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:17:56.6285990Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:17:56.6286458Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:17:56.6286796Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:17:56.6287268Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:17:56.6287597Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:17:56.6288029Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:17:56.6288380Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:17:56.6288705Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:17:56.6289171Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:17:56.6289495Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:17:56.6289906Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:17:56.6290291Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:17:56.6290743Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-12-01T10:17:56.6291344Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:17:56.6291871Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:17:56.6292199Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:17:56.6292625Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:17:56.6292972Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:17:56.6293290Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:17:56.6293785Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:17:56.6294310Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:17:56.6294884Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:17:56.6295265Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:17:56.6295617Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:17:56.6295945Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:17:56.6296279Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:17:56.6296539Z dist init r=1, world=4
2022-12-01T10:17:56.6296708Z dist init r=2, world=4
2022-12-01T10:17:56.6296884Z dist init r=3, world=4
2022-12-01T10:17:56.6297059Z dist init r=0, world=4
2022-12-01T10:17:56.6297249Z skip: Need at least 4 CUDA devices (1.418s)
2022-12-01T10:17:56.6297699Z   test_fsdp_tp_integration_tensor_parallel_size_2_cpu_offload_CPUOffload(offload_params=True) (__main__.TestTPFSDPIntegration)
2022-12-01T10:17:56.6298270Z Tests training for TP + FSDP integration by comparing an FSDP-only ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16401
2022-12-01T10:17:56.6298670Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16402
2022-12-01T10:17:56.6298987Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16403
2022-12-01T10:17:56.6299303Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 16404
2022-12-01T10:17:56.6299758Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:17:56.6300076Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:17:56.6300674Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:17:56.6301026Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:17:56.6301405Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:17:56.6301864Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:17:56.6302194Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:17:56.6302618Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:17:56.6302962Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:17:56.6303281Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:17:56.6303743Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:17:56.6304064Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:17:56.6304555Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:17:56.6304949Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:17:56.6305289Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:17:56.6305791Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:17:56.6306270Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:17:56.6306595Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:17:56.6307020Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:17:56.6307348Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:17:56.6307682Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-12-01T10:17:56.6308178Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:17:56.6308697Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:17:56.6309199Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:17:56.6309585Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:17:56.6309936Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:17:56.6310275Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:17:56.6310606Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:17:56.6310857Z dist init r=2, world=4
2022-12-01T10:17:56.6311037Z dist init r=1, world=4
2022-12-01T10:17:56.6311202Z dist init r=3, world=4
2022-12-01T10:17:56.6311376Z dist init r=0, world=4
2022-12-01T10:17:56.6311578Z skip: Need at least 4 CUDA devices (1.518s)
2022-12-01T10:17:56.6311910Z   test_fsdp_tp_integration_tensor_parallel_size_4_cpu_offload_CPUOffload(offload_params=False) (__main__.TestTPFSDPIntegration)
2022-12-01T10:17:56.6312474Z Tests training for TP + FSDP integration by comparing an FSDP-only ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16457
2022-12-01T10:17:56.6312871Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16458
2022-12-01T10:17:56.6313196Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16459
2022-12-01T10:17:56.6313505Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 16460
2022-12-01T10:17:56.6313951Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:17:56.6314283Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:17:56.6314706Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:17:56.6315039Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:17:56.6315373Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:17:56.6315846Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:17:56.6316159Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:17:56.6316587Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:17:56.6316969Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:17:56.6317303Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:17:56.6317793Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:17:56.6318119Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:17:56.6318547Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:17:56.6318880Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:17:56.6319215Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:17:56.6319708Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:17:56.6320201Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:17:56.6320519Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:17:56.6320942Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:17:56.6321283Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:17:56.6321613Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-12-01T10:17:56.6322090Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:17:56.6322608Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:17:56.6323121Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:17:56.6323502Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:17:56.6323837Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:17:56.6324172Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:17:56.6324516Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:17:56.6324760Z dist init r=3, world=4
2022-12-01T10:17:56.6324942Z dist init r=2, world=4
2022-12-01T10:17:56.6325116Z dist init r=1, world=4
2022-12-01T10:17:56.6325279Z dist init r=0, world=4
2022-12-01T10:17:56.6325483Z skip: Need at least 4 CUDA devices (1.418s)
2022-12-01T10:17:56.6325827Z   test_fsdp_tp_integration_tensor_parallel_size_4_cpu_offload_CPUOffload(offload_params=True) (__main__.TestTPFSDPIntegration)
2022-12-01T10:17:56.6326395Z Tests training for TP + FSDP integration by comparing an FSDP-only ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16513
2022-12-01T10:17:56.6326782Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16514
2022-12-01T10:17:56.6327112Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16515
2022-12-01T10:17:56.6327434Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 16516
2022-12-01T10:17:56.6327883Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:17:56.6328214Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:17:56.6328640Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:17:56.6328982Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:17:56.6329308Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:17:56.6329832Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:17:56.6330198Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:17:56.6330612Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:17:56.6330951Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:17:56.6331288Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:17:56.6331779Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:17:56.6332262Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:17:56.6332589Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:17:56.6333016Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:17:56.6333364Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:17:56.6333684Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:17:56.6334142Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:17:56.6334465Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:17:56.6334879Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:17:56.6335217Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:17:56.6335546Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-12-01T10:17:56.6336033Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:17:56.6336545Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:17:56.6337063Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:17:56.6337521Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:17:56.6337868Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:17:56.6338198Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:17:56.6338531Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:17:56.6338787Z dist init r=3, world=4
2022-12-01T10:17:56.6338955Z dist init r=2, world=4
2022-12-01T10:17:56.6339134Z dist init r=0, world=4
2022-12-01T10:17:56.6339313Z dist init r=1, world=4
2022-12-01T10:17:56.6339506Z skip: Need at least 4 CUDA devices (1.418s)
2022-12-01T10:17:56.6339645Z 
2022-12-01T10:17:56.6339850Z ----------------------------------------------------------------------
2022-12-01T10:17:56.6340090Z Ran 5 tests in 7.429s
2022-12-01T10:17:56.6340203Z 
2022-12-01T10:17:56.6340272Z OK (skipped=5)
2022-12-01T10:17:56.6340506Z 
2022-12-01T10:17:56.6340601Z Generating XML reports...
2022-12-01T10:17:56.6341089Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_tp_integration/TEST-TestTPFSDPIntegration-20221201101748.xml
2022-12-01T10:17:56.6341371Z 
2022-12-01T10:17:56.6341789Z ##[endgroup]
2022-12-01T10:17:56.6342256Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_tp_integration (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_tp_integration_gyh31wac)
2022-12-01T10:17:56.6342537Z 
2022-12-01T10:17:56.6342746Z Running distributed/fsdp/test_fsdp_clip_grad_norm ... [2022-12-01 10:17:56.624142]
2022-12-01T10:17:56.6343358Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_clip_grad_norm.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-12-01 10:17:56.624556]
2022-12-01T10:18:20.9308365Z 
2022-12-01T10:18:20.9310105Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_clip_grad_norm
2022-12-01T10:18:20.9316212Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_clip_grad_norm (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_clip_grad_norm_248zo_bi)
2022-12-01T10:18:20.9316850Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_clip_grad_norm
2022-12-01T10:18:20.9317051Z 
2022-12-01T10:18:20.9317129Z Running tests...
2022-12-01T10:18:20.9317436Z ----------------------------------------------------------------------
2022-12-01T10:18:20.9317770Z   test_fsdp_calc_grad_norm_norm_type_1_3_nested_fsdp_False (__main__.TestCalcuGradNorm)
2022-12-01T10:18:20.9318143Z Test grad norm cal API. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16580
2022-12-01T10:18:20.9318514Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16581
2022-12-01T10:18:20.9319019Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16582
2022-12-01T10:18:20.9319439Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 16583
2022-12-01T10:18:20.9319903Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:18:20.9320243Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:18:20.9320678Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:18:20.9321015Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:18:20.9321361Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:18:20.9321836Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:18:20.9322289Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:18:20.9322709Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:18:20.9323058Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:18:20.9323398Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:18:20.9323898Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:18:20.9324383Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:18:20.9324715Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:18:20.9325144Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:18:20.9325480Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:18:20.9325815Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-12-01T10:18:20.9326287Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:18:20.9326617Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:18:20.9327028Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:18:20.9327377Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:18:20.9327717Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:18:20.9328407Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:18:20.9328981Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:18:20.9329501Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:18:20.9329892Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:18:20.9330233Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:18:20.9330584Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:18:20.9330929Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:18:20.9331193Z dist init r=2, world=4
2022-12-01T10:18:20.9331365Z dist init r=3, world=4
2022-12-01T10:18:20.9331544Z dist init r=1, world=4
2022-12-01T10:18:20.9331725Z dist init r=0, world=4
2022-12-01T10:18:20.9331923Z skip: Need at least 2 CUDA devices (1.655s)
2022-12-01T10:18:20.9332215Z   test_fsdp_calc_grad_norm_norm_type_1_3_nested_fsdp_True (__main__.TestCalcuGradNorm)
2022-12-01T10:18:20.9332578Z Test grad norm cal API. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16636
2022-12-01T10:18:20.9332924Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16637
2022-12-01T10:18:20.9333257Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16638
2022-12-01T10:18:20.9333584Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 16639
2022-12-01T10:18:20.9334037Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:18:20.9334357Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:18:20.9334790Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:18:20.9335179Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:18:20.9335504Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:18:20.9335974Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:18:20.9336305Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:18:20.9336733Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:18:20.9337064Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:18:20.9337508Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:18:20.9337987Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:18:20.9338319Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:18:20.9340759Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:18:20.9341281Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:18:20.9341751Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:18:20.9342427Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:18:20.9342904Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:18:20.9343527Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:18:20.9344024Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:18:20.9344662Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-12-01T10:18:20.9345523Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:18:20.9346355Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:18:20.9347258Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:18:20.9348101Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:18:20.9348687Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:18:20.9349258Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:18:20.9349792Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:18:20.9350414Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:18:20.9350835Z dist init r=1, world=4
2022-12-01T10:18:20.9351126Z dist init r=3, world=4
2022-12-01T10:18:20.9351402Z dist init r=0, world=4
2022-12-01T10:18:20.9351706Z dist init r=2, world=4
2022-12-01T10:18:20.9352054Z skip: Need at least 2 CUDA devices (1.417s)
2022-12-01T10:18:20.9352512Z   test_fsdp_calc_grad_norm_norm_type_2_0_nested_fsdp_False (__main__.TestCalcuGradNorm)
2022-12-01T10:18:20.9353115Z Test grad norm cal API. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16692
2022-12-01T10:18:20.9353693Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16693
2022-12-01T10:18:20.9354242Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16694
2022-12-01T10:18:20.9354818Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 16695
2022-12-01T10:18:20.9355567Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:18:20.9356114Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:18:20.9356818Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:18:20.9357318Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:18:20.9357827Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:18:20.9358505Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:18:20.9358944Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:18:20.9359610Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:18:20.9360108Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:18:20.9360607Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:18:20.9361334Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:18:20.9361841Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:18:20.9362529Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:18:20.9363046Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:18:20.9363579Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-12-01T10:18:20.9364348Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:18:20.9365046Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:18:20.9365867Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:18:20.9366483Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:18:20.9367084Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:18:20.9367893Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:18:20.9368795Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:18:20.9369715Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:18:20.9370568Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:18:20.9371187Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:18:20.9371764Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:18:20.9372325Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:18:20.9372916Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:18:20.9373314Z dist init r=1, world=4
2022-12-01T10:18:20.9373606Z dist init r=3, world=4
2022-12-01T10:18:20.9373926Z dist init r=0, world=4
2022-12-01T10:18:20.9374233Z dist init r=2, world=4
2022-12-01T10:18:20.9374583Z skip: Need at least 2 CUDA devices (1.417s)
2022-12-01T10:18:20.9375090Z   test_fsdp_calc_grad_norm_norm_type_2_0_nested_fsdp_True (__main__.TestCalcuGradNorm)
2022-12-01T10:18:20.9375671Z Test grad norm cal API. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16748
2022-12-01T10:18:20.9376269Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16749
2022-12-01T10:18:20.9376833Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16750
2022-12-01T10:18:20.9377429Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 16751
2022-12-01T10:18:20.9378247Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:18:20.9378835Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:18:20.9379610Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:18:20.9380218Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:18:20.9380967Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:18:20.9381838Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:18:20.9382448Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:18:20.9383209Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:18:20.9383825Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:18:20.9384420Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:18:20.9385253Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:18:20.9385836Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:18:20.9386554Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:18:20.9387297Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:18:20.9387870Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-12-01T10:18:20.9388838Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:18:20.9389424Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:18:20.9390186Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:18:20.9390786Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:18:20.9391382Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:18:20.9392278Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:18:20.9393164Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:18:20.9394076Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:18:20.9394975Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:18:20.9395642Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:18:20.9396265Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:18:20.9396865Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:18:20.9397306Z dist init r=3, world=4
2022-12-01T10:18:20.9397588Z dist init r=0, world=4
2022-12-01T10:18:20.9398032Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:18:20.9398492Z dist init r=2, world=4
2022-12-01T10:18:20.9398786Z dist init r=1, world=4
2022-12-01T10:18:20.9399122Z skip: Need at least 2 CUDA devices (1.417s)
2022-12-01T10:18:20.9399609Z   test_fsdp_calc_grad_norm_norm_type_2_5_nested_fsdp_False (__main__.TestCalcuGradNorm)
2022-12-01T10:18:20.9400261Z Test grad norm cal API. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16804
2022-12-01T10:18:20.9400865Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16805
2022-12-01T10:18:20.9401452Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16806
2022-12-01T10:18:20.9402022Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 16807
2022-12-01T10:18:20.9402837Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:18:20.9403414Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:18:20.9404211Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:18:20.9404854Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:18:20.9405404Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:18:20.9406261Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:18:20.9406858Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:18:20.9407624Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:18:20.9408196Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:18:20.9408780Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:18:20.9409766Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:18:20.9410321Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:18:20.9411137Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:18:20.9411731Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:18:20.9412267Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:18:20.9413091Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:18:20.9413640Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:18:20.9414342Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:18:20.9414902Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:18:20.9415489Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-12-01T10:18:20.9416366Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:18:20.9417359Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:18:20.9418260Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:18:20.9419182Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:18:20.9419852Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:18:20.9420575Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:18:20.9421188Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:18:20.9421816Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:18:20.9422270Z dist init r=0, world=4
2022-12-01T10:18:20.9422559Z dist init r=2, world=4
2022-12-01T10:18:20.9422856Z dist init r=1, world=4
2022-12-01T10:18:20.9423144Z dist init r=3, world=4
2022-12-01T10:18:20.9423513Z skip: Need at least 2 CUDA devices (1.517s)
2022-12-01T10:18:20.9424008Z   test_fsdp_calc_grad_norm_norm_type_2_5_nested_fsdp_True (__main__.TestCalcuGradNorm)
2022-12-01T10:18:20.9424653Z Test grad norm cal API. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16860
2022-12-01T10:18:20.9425259Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16861
2022-12-01T10:18:20.9425803Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16862
2022-12-01T10:18:20.9426369Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 16863
2022-12-01T10:18:20.9427227Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:18:20.9427820Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:18:20.9428598Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:18:20.9429233Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:18:20.9429810Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:18:20.9430591Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:18:20.9431175Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:18:20.9431948Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:18:20.9432723Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:18:20.9433379Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:18:20.9434285Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:18:20.9434863Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:18:20.9435609Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:18:20.9436240Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:18:20.9436829Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-12-01T10:18:20.9437742Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:18:20.9438622Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:18:20.9439208Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:18:20.9439987Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:18:20.9440610Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:18:20.9441180Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:18:20.9442073Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:18:20.9442987Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:18:20.9443908Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:18:20.9444547Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:18:20.9445169Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:18:20.9445777Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:18:20.9446366Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:18:20.9446816Z dist init r=1, world=4
2022-12-01T10:18:20.9447121Z dist init r=0, world=4
2022-12-01T10:18:20.9447409Z dist init r=3, world=4
2022-12-01T10:18:20.9447710Z dist init r=2, world=4
2022-12-01T10:18:20.9448066Z skip: Need at least 2 CUDA devices (1.417s)
2022-12-01T10:18:20.9448554Z   test_fsdp_calc_grad_norm_norm_type_inf_nested_fsdp_False (__main__.TestCalcuGradNorm)
2022-12-01T10:18:20.9449181Z Test grad norm cal API. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16916
2022-12-01T10:18:20.9449807Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16917
2022-12-01T10:18:20.9450377Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16918
2022-12-01T10:18:20.9450921Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 16919
2022-12-01T10:18:20.9451749Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:18:20.9452327Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:18:20.9453097Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:18:20.9453694Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:18:20.9454284Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:18:20.9455258Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:18:20.9455828Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:18:20.9456685Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:18:20.9457360Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:18:20.9457943Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:18:20.9458751Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:18:20.9459320Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:18:20.9460056Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:18:20.9460796Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:18:20.9461347Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-12-01T10:18:20.9462197Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:18:20.9462785Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:18:20.9463541Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:18:20.9464167Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:18:20.9464734Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:18:20.9465607Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:18:20.9466524Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:18:20.9467480Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:18:20.9468419Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:18:20.9469122Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:18:20.9469735Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:18:20.9470359Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:18:20.9470990Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:18:20.9471439Z dist init r=1, world=4
2022-12-01T10:18:20.9471759Z dist init r=0, world=4
2022-12-01T10:18:20.9472083Z dist init r=3, world=4
2022-12-01T10:18:20.9472382Z dist init r=2, world=4
2022-12-01T10:18:20.9472725Z skip: Need at least 2 CUDA devices (1.517s)
2022-12-01T10:18:20.9473207Z   test_fsdp_calc_grad_norm_norm_type_inf_nested_fsdp_True (__main__.TestCalcuGradNorm)
2022-12-01T10:18:20.9473828Z Test grad norm cal API. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16972
2022-12-01T10:18:20.9474432Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16973
2022-12-01T10:18:20.9475017Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16974
2022-12-01T10:18:20.9475579Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 16975
2022-12-01T10:18:20.9476386Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:18:20.9476979Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:18:20.9477758Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:18:20.9478538Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:18:20.9479172Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:18:20.9480019Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:18:20.9480590Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:18:20.9481342Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:18:20.9481953Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:18:20.9482507Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:18:20.9483328Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:18:20.9483868Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:18:20.9484588Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:18:20.9485163Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:18:20.9485739Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:18:20.9486624Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:18:20.9487528Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:18:20.9488043Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:18:20.9488765Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:18:20.9489373Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:18:20.9489936Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-12-01T10:18:20.9490726Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:18:20.9491601Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:18:20.9492515Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:18:20.9493191Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:18:20.9493760Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:18:20.9494303Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:18:20.9494940Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:18:20.9495397Z dist init r=2, world=4
2022-12-01T10:18:20.9495677Z dist init r=0, world=4
2022-12-01T10:18:20.9495975Z dist init r=1, world=4
2022-12-01T10:18:20.9496276Z dist init r=3, world=4
2022-12-01T10:18:20.9496605Z skip: Need at least 2 CUDA devices (1.417s)
2022-12-01T10:18:20.9497187Z   test_fsdp_clip_grad_norm_norm_type_2_0_nested_fsdp_False_cpu_offload_CPUOffload(offload_params=False) (__main__.TestClipGradNorm)
2022-12-01T10:18:20.9497979Z Test FSDP with clip grad norm. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17028
2022-12-01T10:18:20.9498575Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17029
2022-12-01T10:18:20.9499126Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17030
2022-12-01T10:18:20.9499701Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 17031
2022-12-01T10:18:20.9500796Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:18:20.9501416Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:18:20.9502186Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:18:20.9502802Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:18:20.9503360Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:18:20.9504196Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:18:20.9504762Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:18:20.9505535Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:18:20.9506138Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:18:20.9506707Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:18:20.9507568Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:18:20.9508117Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:18:20.9508868Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:18:20.9509480Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:18:20.9510066Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-12-01T10:18:20.9510904Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:18:20.9511467Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:18:20.9512230Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:18:20.9512842Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:18:20.9513424Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:18:20.9514332Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:18:20.9515295Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:18:20.9516236Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:18:20.9517145Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:18:20.9517827Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:18:20.9518445Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:18:20.9519063Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:18:20.9519629Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:18:20.9520077Z dist init r=2, world=4
2022-12-01T10:18:20.9520388Z dist init r=1, world=4
2022-12-01T10:18:20.9520669Z dist init r=0, world=4
2022-12-01T10:18:20.9520968Z dist init r=3, world=4
2022-12-01T10:18:20.9521336Z skip: Need at least 2 CUDA devices (1.417s)
2022-12-01T10:18:20.9521906Z   test_fsdp_clip_grad_norm_norm_type_2_0_nested_fsdp_False_cpu_offload_CPUOffload(offload_params=True) (__main__.TestClipGradNorm)
2022-12-01T10:18:20.9522738Z Test FSDP with clip grad norm. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17084
2022-12-01T10:18:20.9523400Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17085
2022-12-01T10:18:20.9524081Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17086
2022-12-01T10:18:20.9524648Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 17087
2022-12-01T10:18:20.9525418Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:18:20.9525949Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:18:20.9526698Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:18:20.9527275Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:18:20.9527848Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:18:20.9528709Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:18:20.9529287Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:18:20.9530080Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:18:20.9530667Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:18:20.9531209Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:18:20.9532013Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:18:20.9532590Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:18:20.9533365Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:18:20.9533931Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:18:20.9534476Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:18:20.9535324Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:18:20.9535898Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:18:20.9536640Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:18:20.9537344Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:18:20.9537935Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-12-01T10:18:20.9538836Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:18:20.9539704Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:18:20.9540718Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:18:20.9541624Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:18:20.9542300Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:18:20.9542894Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:18:20.9543499Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:18:20.9544101Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:18:20.9544539Z dist init r=2, world=4
2022-12-01T10:18:20.9544862Z dist init r=0, world=4
2022-12-01T10:18:20.9545169Z dist init r=1, world=4
2022-12-01T10:18:20.9545610Z dist init r=3, world=4
2022-12-01T10:18:20.9545965Z skip: Need at least 2 CUDA devices (1.417s)
2022-12-01T10:18:20.9546629Z   test_fsdp_clip_grad_norm_norm_type_2_0_nested_fsdp_True_cpu_offload_CPUOffload(offload_params=False) (__main__.TestClipGradNorm)
2022-12-01T10:18:20.9547372Z Test FSDP with clip grad norm. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17140
2022-12-01T10:18:20.9547976Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17141
2022-12-01T10:18:20.9548575Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17142
2022-12-01T10:18:20.9549156Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 17143
2022-12-01T10:18:20.9549968Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:18:20.9550529Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:18:20.9551307Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:18:20.9551912Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:18:20.9552496Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:18:20.9553310Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:18:20.9553872Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:18:20.9554620Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:18:20.9555188Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:18:20.9555728Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:18:20.9556601Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:18:20.9557139Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:18:20.9557897Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:18:20.9558522Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:18:20.9559127Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:18:20.9559996Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:18:20.9560875Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:18:20.9561427Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:18:20.9562181Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:18:20.9562731Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:18:20.9563283Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-12-01T10:18:20.9564179Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:18:20.9565050Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:18:20.9565959Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:18:20.9566625Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:18:20.9567227Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:18:20.9567946Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:18:20.9568629Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:18:20.9569067Z dist init r=1, world=4
2022-12-01T10:18:20.9569337Z dist init r=3, world=4
2022-12-01T10:18:20.9569630Z dist init r=0, world=4
2022-12-01T10:18:20.9569924Z dist init r=2, world=4
2022-12-01T10:18:20.9570246Z skip: Need at least 2 CUDA devices (1.417s)
2022-12-01T10:18:20.9570828Z   test_fsdp_clip_grad_norm_norm_type_2_0_nested_fsdp_True_cpu_offload_CPUOffload(offload_params=True) (__main__.TestClipGradNorm)
2022-12-01T10:18:20.9571541Z Test FSDP with clip grad norm. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17196
2022-12-01T10:18:20.9572175Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17197
2022-12-01T10:18:20.9572763Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17198
2022-12-01T10:18:20.9573330Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 17199
2022-12-01T10:18:20.9574159Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:18:20.9574740Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:18:20.9575492Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:18:20.9576108Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:18:20.9576638Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:18:20.9577538Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:18:20.9578108Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:18:20.9578864Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:18:20.9579464Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:18:20.9580016Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:18:20.9581300Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:18:20.9581686Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:18:20.9582120Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:18:20.9582531Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:18:20.9582892Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:18:20.9583411Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:18:20.9583904Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:18:20.9584247Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:18:20.9584684Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:18:20.9585034Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:18:20.9585358Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-12-01T10:18:20.9585860Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:18:20.9586389Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:18:20.9587043Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:18:20.9587521Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:18:20.9587875Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:18:20.9588232Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:18:20.9588564Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:18:20.9588822Z dist init r=3, world=4
2022-12-01T10:18:20.9589013Z dist init r=0, world=4
2022-12-01T10:18:20.9589182Z dist init r=1, world=4
2022-12-01T10:18:20.9589365Z dist init r=2, world=4
2022-12-01T10:18:20.9589579Z skip: Need at least 2 CUDA devices (1.417s)
2022-12-01T10:18:20.9589912Z   test_fsdp_clip_grad_norm_norm_type_inf_nested_fsdp_False_cpu_offload_CPUOffload(offload_params=False) (__main__.TestClipGradNorm)
2022-12-01T10:18:20.9590343Z Test FSDP with clip grad norm. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17252
2022-12-01T10:18:20.9590715Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17253
2022-12-01T10:18:20.9591048Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17254
2022-12-01T10:18:20.9591362Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 17255
2022-12-01T10:18:20.9591819Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:18:20.9592156Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:18:20.9592592Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:18:20.9592930Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:18:20.9593386Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:18:20.9593882Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:18:20.9594205Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:18:20.9594637Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:18:20.9594984Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:18:20.9595326Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:18:20.9595786Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:18:20.9596118Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:18:20.9596555Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:18:20.9596902Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:18:20.9597231Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-12-01T10:18:20.9597731Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:18:20.9598233Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:18:20.9598547Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:18:20.9599028Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:18:20.9599382Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:18:20.9599723Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:18:20.9600286Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:18:20.9600875Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:18:20.9601399Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:18:20.9601790Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:18:20.9602131Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:18:20.9602476Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:18:20.9602827Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:18:20.9603086Z dist init r=1, world=4
2022-12-01T10:18:20.9603274Z dist init r=0, world=4
2022-12-01T10:18:20.9603463Z dist init r=3, world=4
2022-12-01T10:18:20.9603639Z dist init r=2, world=4
2022-12-01T10:18:20.9603849Z skip: Need at least 2 CUDA devices (1.417s)
2022-12-01T10:18:20.9604193Z   test_fsdp_clip_grad_norm_norm_type_inf_nested_fsdp_False_cpu_offload_CPUOffload(offload_params=True) (__main__.TestClipGradNorm)
2022-12-01T10:18:20.9604616Z Test FSDP with clip grad norm. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17308
2022-12-01T10:18:20.9604974Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17309
2022-12-01T10:18:20.9605309Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17310
2022-12-01T10:18:20.9605637Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 17311
2022-12-01T10:18:20.9606093Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:18:20.9606433Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:18:20.9606876Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:18:20.9607229Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:18:20.9607602Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:18:20.9608084Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:18:20.9608417Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:18:20.9608858Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:18:20.9609196Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:18:20.9609540Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:18:20.9610020Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:18:20.9610342Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:18:20.9610780Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:18:20.9611125Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:18:20.9611466Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:18:20.9611927Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:18:20.9612260Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:18:20.9612686Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:18:20.9613060Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:18:20.9613452Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-12-01T10:18:20.9613954Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:18:20.9614502Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:18:20.9615025Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:18:20.9615548Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:18:20.9615924Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:18:20.9616280Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:18:20.9616625Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:18:20.9616982Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:18:20.9617310Z dist init r=3, world=4
2022-12-01T10:18:20.9617504Z dist init r=1, world=4
2022-12-01T10:18:20.9617685Z dist init r=0, world=4
2022-12-01T10:18:20.9617852Z dist init r=2, world=4
2022-12-01T10:18:20.9618068Z skip: Need at least 2 CUDA devices (1.316s)
2022-12-01T10:18:20.9618415Z   test_fsdp_clip_grad_norm_norm_type_inf_nested_fsdp_True_cpu_offload_CPUOffload(offload_params=False) (__main__.TestClipGradNorm)
2022-12-01T10:18:20.9618819Z Test FSDP with clip grad norm. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17364
2022-12-01T10:18:20.9619189Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17365
2022-12-01T10:18:20.9619527Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17366
2022-12-01T10:18:20.9619854Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 17367
2022-12-01T10:18:20.9620313Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:18:20.9620973Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:18:20.9621613Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:18:20.9622097Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:18:20.9622441Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:18:20.9622917Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:18:20.9623251Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:18:20.9623679Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:18:20.9624030Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:18:20.9624371Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:18:20.9624841Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:18:20.9625154Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:18:20.9625582Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:18:20.9625926Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:18:20.9637331Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-12-01T10:18:20.9638021Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:18:20.9638393Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:18:20.9638835Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:18:20.9639187Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:18:20.9639524Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:18:20.9640013Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:18:20.9640538Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:18:20.9641051Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:18:20.9641567Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:18:20.9641939Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:18:20.9642291Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:18:20.9642634Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:18:20.9642966Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:18:20.9643228Z dist init r=2, world=4
2022-12-01T10:18:20.9643410Z dist init r=3, world=4
2022-12-01T10:18:20.9643579Z dist init r=0, world=4
2022-12-01T10:18:20.9643757Z dist init r=1, world=4
2022-12-01T10:18:20.9643966Z skip: Need at least 2 CUDA devices (1.417s)
2022-12-01T10:18:20.9644307Z   test_fsdp_clip_grad_norm_norm_type_inf_nested_fsdp_True_cpu_offload_CPUOffload(offload_params=True) (__main__.TestClipGradNorm)
2022-12-01T10:18:20.9644710Z Test FSDP with clip grad norm. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17420
2022-12-01T10:18:20.9645076Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17421
2022-12-01T10:18:20.9645411Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17422
2022-12-01T10:18:20.9645729Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 17423
2022-12-01T10:18:20.9646189Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:18:20.9646521Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:18:20.9646952Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:18:20.9647287Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:18:20.9647630Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:18:20.9648104Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:18:20.9648431Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:18:20.9648847Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:18:20.9649195Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:18:20.9649530Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:18:20.9649983Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:18:20.9650311Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:18:20.9650774Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:18:20.9651148Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:18:20.9651468Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:18:20.9651940Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:18:20.9652271Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:18:20.9652681Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:18:20.9653022Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:18:20.9653358Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-12-01T10:18:20.9653854Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:18:20.9654360Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:18:20.9654872Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:18:20.9655391Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:18:20.9655775Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:18:20.9656115Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:18:20.9656458Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:18:20.9656802Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:18:20.9657051Z dist init r=0, world=4
2022-12-01T10:18:20.9657322Z dist init r=2, world=4
2022-12-01T10:18:20.9657517Z dist init r=1, world=4
2022-12-01T10:18:20.9657694Z dist init r=3, world=4
2022-12-01T10:18:20.9657886Z skip: Need at least 2 CUDA devices (1.417s)
2022-12-01T10:18:20.9658025Z 
2022-12-01T10:18:20.9658228Z ----------------------------------------------------------------------
2022-12-01T10:18:20.9658473Z Ran 16 tests in 23.010s
2022-12-01T10:18:20.9658575Z 
2022-12-01T10:18:20.9658694Z OK (skipped=16)
2022-12-01T10:18:20.9658806Z 
2022-12-01T10:18:20.9658890Z Generating XML reports...
2022-12-01T10:18:20.9659353Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_clip_grad_norm/TEST-TestCalcuGradNorm-20221201101757.xml
2022-12-01T10:18:20.9659940Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_clip_grad_norm/TEST-TestClipGradNorm-20221201101757.xml
2022-12-01T10:18:20.9660209Z 
2022-12-01T10:18:20.9660727Z ##[endgroup]
2022-12-01T10:18:20.9661213Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_clip_grad_norm (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_clip_grad_norm_248zo_bi)
2022-12-01T10:18:20.9661490Z 
2022-12-01T10:18:20.9661731Z Running distributed/fsdp/test_distributed_checkpoint ... [2022-12-01 10:18:20.931722]
2022-12-01T10:18:20.9662269Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_distributed_checkpoint.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-12-01 10:18:20.932098]
2022-12-01T10:18:25.1273230Z 
2022-12-01T10:18:25.1273835Z Expand the folded group to see the log file of distributed/fsdp/test_distributed_checkpoint
2022-12-01T10:18:25.1275191Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_distributed_checkpoint (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_distributed_checkpoint_e8qvuf_x)
2022-12-01T10:18:25.1276217Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_distributed_checkpoint
2022-12-01T10:18:25.1276642Z 
2022-12-01T10:18:25.1276723Z Running tests...
2022-12-01T10:18:25.1277109Z ----------------------------------------------------------------------
2022-12-01T10:18:25.1277760Z   test_distributed_checkpoint_state_dict_type_StateDictType_LOCAL_STATE_DICT (__main__.TestDistributedCheckpoint) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17487
2022-12-01T10:18:25.1278350Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17488
2022-12-01T10:18:25.1279174Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:18:25.1279516Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:18:25.1279952Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:18:25.1280293Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:18:25.1280733Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:18:25.1281296Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:18:25.1281834Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:18:25.1282470Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:18:25.1282818Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:18:25.1283152Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:18:25.1283632Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:18:25.1284158Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:18:25.1284547Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:18:25.1284903Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:18:25.1285154Z dist init r=1, world=2
2022-12-01T10:18:25.1285338Z dist init r=0, world=2
2022-12-01T10:18:25.1285551Z skip: Need at least 2 CUDA devices (1.557s)
2022-12-01T10:18:25.1285961Z   test_distributed_checkpoint_state_dict_type_StateDictType_SHARDED_STATE_DICT (__main__.TestDistributedCheckpoint) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17515
2022-12-01T10:18:25.1286414Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17516
2022-12-01T10:18:25.1286869Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:18:25.1287201Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:18:25.1287621Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:18:25.1287971Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:18:25.1288311Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:18:25.1288782Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:18:25.1289097Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:18:25.1289526Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:18:25.1289872Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:18:25.1290193Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:18:25.1290756Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:18:25.1291322Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:18:25.1291711Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:18:25.1292048Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:18:25.1292307Z dist init r=1, world=2
2022-12-01T10:18:25.1292491Z dist init r=0, world=2
2022-12-01T10:18:25.1292688Z skip: Need at least 2 CUDA devices (1.311s)
2022-12-01T10:18:25.1292826Z 
2022-12-01T10:18:25.1293024Z ----------------------------------------------------------------------
2022-12-01T10:18:25.1293267Z Ran 2 tests in 2.869s
2022-12-01T10:18:25.1293382Z 
2022-12-01T10:18:25.1293454Z OK (skipped=2)
2022-12-01T10:18:25.1293549Z 
2022-12-01T10:18:25.1293635Z Generating XML reports...
2022-12-01T10:18:25.1294144Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_distributed_checkpoint/TEST-TestDistributedCheckpoint-20221201101821.xml
2022-12-01T10:18:25.1294441Z 
2022-12-01T10:18:25.1294713Z ##[endgroup]
2022-12-01T10:18:25.1295194Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_distributed_checkpoint (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_distributed_checkpoint_e8qvuf_x)
2022-12-01T10:18:25.1295484Z 
2022-12-01T10:18:25.1295690Z Running distributed/fsdp/test_fsdp_uneven ... [2022-12-01 10:18:25.127658]
2022-12-01T10:18:25.1296208Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_uneven.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-12-01 10:18:25.128004]
2022-12-01T10:18:28.0959146Z 
2022-12-01T10:18:28.0959767Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_uneven
2022-12-01T10:18:28.0960490Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_uneven (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_uneven_2kroik_s)
2022-12-01T10:18:28.0961689Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_uneven
2022-12-01T10:18:28.0962088Z 
2022-12-01T10:18:28.0962176Z Running tests...
2022-12-01T10:18:28.0962519Z ----------------------------------------------------------------------
2022-12-01T10:18:28.0963043Z   test_one_iteration (__main__.TestUnevenParamShard)
2022-12-01T10:18:28.0963537Z Test FSDP with uneven divide of parameter shards. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17554
2022-12-01T10:18:28.0981410Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17555
2022-12-01T10:18:28.0982117Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17556
2022-12-01T10:18:28.0982661Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 17557
2022-12-01T10:18:28.0983576Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:18:28.0983958Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:18:28.0984398Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:18:28.0984761Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:18:28.0985098Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:18:28.0985558Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:18:28.0985887Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:18:28.0986313Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:18:28.0986659Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:18:28.0987141Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:18:28.0987670Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:18:28.0987998Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:18:28.0995970Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:18:28.0996619Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:18:28.0997255Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:18:28.0998002Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:18:28.0998337Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:18:28.0998765Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:18:28.0999177Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:18:28.0999797Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-12-01T10:18:28.1000500Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:18:28.1001072Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:18:28.1001601Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:18:28.1002123Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:18:28.1002501Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:18:28.1002859Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:18:28.1003197Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:18:28.1003541Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:18:28.1003792Z dist init r=0, world=4
2022-12-01T10:18:28.1003976Z dist init r=1, world=4
2022-12-01T10:18:28.1004155Z dist init r=3, world=4
2022-12-01T10:18:28.1004321Z dist init r=2, world=4
2022-12-01T10:18:28.1004541Z skip: Need at least 2 CUDA devices (1.659s)
2022-12-01T10:18:28.1004678Z 
2022-12-01T10:18:28.1004885Z ----------------------------------------------------------------------
2022-12-01T10:18:28.1005114Z Ran 1 test in 1.660s
2022-12-01T10:18:28.1005228Z 
2022-12-01T10:18:28.1005303Z OK (skipped=1)
2022-12-01T10:18:28.1005411Z 
2022-12-01T10:18:28.1005497Z Generating XML reports...
2022-12-01T10:18:28.1005964Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_uneven/TEST-TestUnevenParamShard-20221201101826.xml
2022-12-01T10:18:28.1006239Z 
2022-12-01T10:18:28.1006594Z ##[endgroup]
2022-12-01T10:18:28.1007050Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_uneven (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_uneven_2kroik_s)
2022-12-01T10:18:28.1007311Z 
2022-12-01T10:18:28.1007536Z Running distributed/fsdp/test_fsdp_multiple_forward ... [2022-12-01 10:18:28.096235]
2022-12-01T10:18:28.1008082Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_multiple_forward.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-12-01 10:18:28.096907]
2022-12-01T10:18:31.0449008Z 
2022-12-01T10:18:31.0449934Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_multiple_forward
2022-12-01T10:18:31.0451682Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_multiple_forward (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_multiple_forward_ergbpqrv)
2022-12-01T10:18:31.0452858Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_multiple_forward
2022-12-01T10:18:31.0453202Z 
2022-12-01T10:18:31.0453323Z Running tests...
2022-12-01T10:18:31.0453816Z ----------------------------------------------------------------------
2022-12-01T10:18:31.0454432Z   test_multi_forward (__main__.TestMultiForward) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17621
2022-12-01T10:18:31.0455028Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17622
2022-12-01T10:18:31.0455572Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17623
2022-12-01T10:18:31.0456088Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 17624
2022-12-01T10:18:31.0456942Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:18:31.0457436Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:18:31.0457880Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:18:31.0458234Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:18:31.0458562Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:18:31.0459044Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:18:31.0459373Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:18:31.0459802Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:18:31.0460133Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:18:31.0460672Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:18:31.0461152Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:18:31.0461615Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:18:31.0462402Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:18:31.0462872Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:18:31.0463326Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-12-01T10:18:31.0464126Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:18:31.0464455Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:18:31.0464887Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:18:31.0465236Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:18:31.0465562Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:18:31.0466056Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:18:31.0466579Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:18:31.0467094Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:18:31.0467595Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:18:31.0467980Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:18:31.0468442Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:18:31.0468821Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:18:31.0469173Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:18:31.0469432Z dist init r=3, world=4
2022-12-01T10:18:31.0469618Z dist init r=2, world=4
2022-12-01T10:18:31.0469784Z dist init r=1, world=4
2022-12-01T10:18:31.0469967Z dist init r=0, world=4
2022-12-01T10:18:31.0470177Z skip: Need at least 2 CUDA devices (1.658s)
2022-12-01T10:18:31.0470302Z 
2022-12-01T10:18:31.0470505Z ----------------------------------------------------------------------
2022-12-01T10:18:31.0470751Z Ran 1 test in 1.659s
2022-12-01T10:18:31.0470863Z 
2022-12-01T10:18:31.0470935Z OK (skipped=1)
2022-12-01T10:18:31.0471042Z 
2022-12-01T10:18:31.0471114Z Generating XML reports...
2022-12-01T10:18:31.0471586Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_multiple_forward/TEST-TestMultiForward-20221201101829.xml
2022-12-01T10:18:31.0471858Z 
2022-12-01T10:18:31.0472134Z ##[endgroup]
2022-12-01T10:18:31.0472611Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_multiple_forward (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_multiple_forward_ergbpqrv)
2022-12-01T10:18:31.0472899Z 
2022-12-01T10:18:31.0473118Z Running distributed/fsdp/test_fsdp_overlap ... [2022-12-01 10:18:31.042752]
2022-12-01T10:18:31.0473640Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_overlap.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-12-01 10:18:31.043240]
2022-12-01T10:18:33.7845251Z 
2022-12-01T10:18:33.7845781Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_overlap
2022-12-01T10:18:33.7847027Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_overlap (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_overlap_x0nymy_f)
2022-12-01T10:18:33.7847774Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_overlap
2022-12-01T10:18:33.7847994Z 
2022-12-01T10:18:33.7848071Z Running tests...
2022-12-01T10:18:33.7848369Z ----------------------------------------------------------------------
2022-12-01T10:18:33.7848791Z   test_forward_overlap (__main__.TestForwardOverlapWorldSizeOne) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17688
2022-12-01T10:18:33.7849338Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:18:33.7849672Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:18:33.7850090Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:18:33.7850440Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:18:33.7850783Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:18:33.7851275Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-12-01T10:18:33.7851721Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:18:33.7852053Z dist init r=0, world=1
2022-12-01T10:18:33.7852333Z skip: Need at least 2 CUDA devices (1.444s)
2022-12-01T10:18:33.7853278Z   test_forward_overlap (__main__.TestForwardOverlapWorldSizeTwo) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/71183 for allplatform(s) . If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.001s)
2022-12-01T10:18:33.7853765Z 
2022-12-01T10:18:33.7853995Z ----------------------------------------------------------------------
2022-12-01T10:18:33.7854445Z Ran 2 tests in 1.446s
2022-12-01T10:18:33.7854673Z 
2022-12-01T10:18:33.7854750Z OK (skipped=2)
2022-12-01T10:18:33.7854860Z 
2022-12-01T10:18:33.7854947Z Generating XML reports...
2022-12-01T10:18:33.7855635Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_overlap/TEST-TestForwardOverlapWorldSizeOne-20221201101832.xml
2022-12-01T10:18:33.7856358Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_overlap/TEST-TestForwardOverlapWorldSizeTwo-20221201101832.xml
2022-12-01T10:18:33.7856647Z 
2022-12-01T10:18:33.7856916Z ##[endgroup]
2022-12-01T10:18:33.7857437Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_overlap (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_overlap_x0nymy_f)
2022-12-01T10:18:33.7857704Z 
2022-12-01T10:18:33.7857934Z Running distributed/fsdp/test_checkpoint_wrapper ... [2022-12-01 10:18:33.785095]
2022-12-01T10:18:33.7858484Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_checkpoint_wrapper.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-12-01 10:18:33.785449]
2022-12-01T10:18:35.3162327Z 
2022-12-01T10:18:35.3162899Z Expand the folded group to see the log file of distributed/fsdp/test_checkpoint_wrapper
2022-12-01T10:18:35.3164176Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_checkpoint_wrapper (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_checkpoint_wrapper_mezagvzb)
2022-12-01T10:18:35.3165196Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_checkpoint_wrapper
2022-12-01T10:18:35.3165418Z 
2022-12-01T10:18:35.3165494Z Running tests...
2022-12-01T10:18:35.3165910Z ----------------------------------------------------------------------
2022-12-01T10:18:35.3166278Z   test_apply_activation_checkpointing (__main__.CheckpointWrapperTest)
2022-12-01T10:18:35.3166643Z Ensures that `apply_activation_checkpointing` can be used ... ok (0.285s)
2022-12-01T10:18:35.3167013Z   test_checkpoint_wrapper_cpu_offload (__main__.CheckpointWrapperTest) ... skip: Test requires CUDA (0.001s)
2022-12-01T10:18:35.3167443Z   test_checkpoint_wrapper_kwarg_support (__main__.CheckpointWrapperTest) ... ok (0.007s)
2022-12-01T10:18:35.3167760Z   test_checkpoint_wrapper_parity (__main__.CheckpointWrapperTest)
2022-12-01T10:18:35.3168139Z Tests that using checkpoint_wrapper or the functional ... skip: Test requires CUDA (0.002s)
2022-12-01T10:18:35.3168478Z   test_forward_missing_attributes (__main__.CheckpointWrapperTest) ... ok (0.001s)
2022-12-01T10:18:35.3168829Z   test_fqn (__main__.CheckpointWrapperTest) ... ok (0.001s)
2022-12-01T10:18:35.3169162Z   test_load_activation_checkpointed_module (__main__.CheckpointWrapperTest) ... ok (0.002s)
2022-12-01T10:18:35.3169362Z 
2022-12-01T10:18:35.3169613Z ----------------------------------------------------------------------
2022-12-01T10:18:35.3169859Z Ran 7 tests in 0.300s
2022-12-01T10:18:35.3169973Z 
2022-12-01T10:18:35.3170032Z OK (skipped=2)
2022-12-01T10:18:35.3170164Z 
2022-12-01T10:18:35.3170280Z Generating XML reports...
2022-12-01T10:18:35.3170775Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_checkpoint_wrapper/TEST-CheckpointWrapperTest-20221201101834.xml
2022-12-01T10:18:35.3171116Z 
2022-12-01T10:18:35.3171337Z ##[endgroup]
2022-12-01T10:18:35.3171872Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_checkpoint_wrapper (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_checkpoint_wrapper_mezagvzb)
2022-12-01T10:18:35.3172152Z 
2022-12-01T10:18:35.3172350Z Running distributed/fsdp/test_utils ... [2022-12-01 10:18:35.316521]
2022-12-01T10:18:35.3172903Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_utils.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-12-01 10:18:35.316849]
2022-12-01T10:18:36.8065066Z 
2022-12-01T10:18:36.8065630Z Expand the folded group to see the log file of distributed/fsdp/test_utils
2022-12-01T10:18:36.8067005Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_utils (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_utils_q6dcumx2)
2022-12-01T10:18:36.8068097Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_utils
2022-12-01T10:18:36.8068421Z 
2022-12-01T10:18:36.8068534Z Running tests...
2022-12-01T10:18:36.8068968Z ----------------------------------------------------------------------
2022-12-01T10:18:36.8069505Z   test_apply_to_tensors_cpu_cuda (__main__.TestUtils) ... skip: Skipped due to lack of GPU (0.239s)
2022-12-01T10:18:36.8070152Z   test_apply_to_tensors_devices_['cpu'] (__main__.TestUtils) ... ok (0.004s)
2022-12-01T10:18:36.8070772Z   test_apply_to_tensors_devices_['cuda'] (__main__.TestUtils) ... skip: Skipped due to lack of GPU (0.002s)
2022-12-01T10:18:36.8071267Z   test_packed_sequence (__main__.TestUtils)
2022-12-01T10:18:36.8071718Z Test to ensure RNN packed sequences are modified correctly. ... ok (0.003s)
2022-12-01T10:18:36.8072156Z   test_replace_by_prefix (__main__.TestUtils) ... ok (0.001s)
2022-12-01T10:18:36.8072378Z 
2022-12-01T10:18:36.8072702Z ----------------------------------------------------------------------
2022-12-01T10:18:36.8073087Z Ran 5 tests in 0.249s
2022-12-01T10:18:36.8073261Z 
2022-12-01T10:18:36.8073376Z OK (skipped=2)
2022-12-01T10:18:36.8073543Z 
2022-12-01T10:18:36.8073661Z Generating XML reports...
2022-12-01T10:18:36.8074343Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_utils/TEST-TestUtils-20221201101836.xml
2022-12-01T10:18:36.8074667Z 
2022-12-01T10:18:36.8075044Z ##[endgroup]
2022-12-01T10:18:36.8075617Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_utils (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_utils_q6dcumx2)
2022-12-01T10:18:36.8075960Z 
2022-12-01T10:18:36.8076305Z Running distributed/algorithms/ddp_comm_hooks/test_ddp_hooks ... [2022-12-01 10:18:36.806854]
2022-12-01T10:18:36.8077118Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/algorithms/ddp_comm_hooks/test_ddp_hooks.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-12-01 10:18:36.807245]
2022-12-01T10:18:37.9507812Z 
2022-12-01T10:18:37.9508351Z Expand the folded group to see the log file of distributed/algorithms/ddp_comm_hooks/test_ddp_hooks
2022-12-01T10:18:37.9509483Z ##[group]PRINTING LOG FILE of distributed/algorithms/ddp_comm_hooks/test_ddp_hooks (/var/lib/jenkins/workspace/test/test-reports/distributed-algorithms-ddp_comm_hooks-test_ddp_hooks_xmy4lle4)
2022-12-01T10:18:37.9510354Z Test results will be stored in test-reports/python-unittest/distributed.algorithms.ddp_comm_hooks.test_ddp_hooks
2022-12-01T10:18:37.9510763Z 
2022-12-01T10:18:37.9510904Z Running tests...
2022-12-01T10:18:37.9511368Z ----------------------------------------------------------------------
2022-12-01T10:18:37.9511727Z   test_ddp_comm_hook_allreduce_hook (__main__.DistributedDataParallelCommHookTest)
2022-12-01T10:18:37.9512223Z This unit test verifies the ``allreduce`` hook registered case gives same result ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-12-01T10:18:37.9512650Z   test_ddp_comm_hook_fp16compress_hook (__main__.DistributedDataParallelCommHookTest)
2022-12-01T10:18:37.9513129Z This unit test verifies the ``fp16 compress`` hook registered case ... skip: c10d was not compiled with the NCCL backend (0.000s)
2022-12-01T10:18:37.9513502Z   test_ddp_comm_hook_noop_hook (__main__.DistributedDataParallelCommHookTest)
2022-12-01T10:18:37.9513898Z This unit test verifies the ``noop`` hook registered case and a subsequent allreduce ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-12-01T10:18:37.9514293Z   test_ddp_comm_hook_quantize_per_channel_hook (__main__.DistributedDataParallelCommHookTest)
2022-12-01T10:18:37.9514695Z This unit test verifies the ``quantize per channel`` hook registered case ... skip: c10d was not compiled with the NCCL backend (0.000s)
2022-12-01T10:18:37.9515151Z   test_ddp_comm_hook_quantize_per_tensor_hook (__main__.DistributedDataParallelCommHookTest)
2022-12-01T10:18:37.9515813Z This unit test verifies the ``quantize per tensor`` hook registered case ... skip: c10d was not compiled with the NCCL backend (0.000s)
2022-12-01T10:18:37.9516354Z   test_is_last_hook (__main__.DistributedDataParallelCommHookTest) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-12-01T10:18:37.9516588Z 
2022-12-01T10:18:37.9516799Z ----------------------------------------------------------------------
2022-12-01T10:18:37.9517044Z Ran 6 tests in 0.004s
2022-12-01T10:18:37.9517159Z 
2022-12-01T10:18:37.9517218Z OK (skipped=6)
2022-12-01T10:18:37.9517327Z 
2022-12-01T10:18:37.9517411Z Generating XML reports...
2022-12-01T10:18:37.9517970Z Generated XML report: test-reports/python-unittest/distributed.algorithms.ddp_comm_hooks.test_ddp_hooks/TEST-DistributedDataParallelCommHookTest-20221201101837.xml
2022-12-01T10:18:37.9518298Z 
2022-12-01T10:18:37.9518546Z ##[endgroup]
2022-12-01T10:18:37.9519054Z FINISHED PRINTING LOG FILE of distributed/algorithms/ddp_comm_hooks/test_ddp_hooks (/var/lib/jenkins/workspace/test/test-reports/distributed-algorithms-ddp_comm_hooks-test_ddp_hooks_xmy4lle4)
2022-12-01T10:18:37.9519356Z 
2022-12-01T10:18:37.9519601Z Running distributed/_shard/checkpoint/test_file_system_checkpoint ... [2022-12-01 10:18:37.951136]
2022-12-01T10:18:37.9520176Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/checkpoint/test_file_system_checkpoint.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-12-01 10:18:37.951465]
2022-12-01T10:18:39.4577876Z 
2022-12-01T10:18:39.4578516Z Expand the folded group to see the log file of distributed/_shard/checkpoint/test_file_system_checkpoint
2022-12-01T10:18:39.4579705Z ##[group]PRINTING LOG FILE of distributed/_shard/checkpoint/test_file_system_checkpoint (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-checkpoint-test_file_system_checkpoint_3zug31e3)
2022-12-01T10:18:39.4580565Z Test results will be stored in test-reports/python-unittest/distributed._shard.checkpoint.test_file_system_checkpoint
2022-12-01T10:18:39.4580805Z 
2022-12-01T10:18:39.4580890Z Running tests...
2022-12-01T10:18:39.4581208Z ----------------------------------------------------------------------
2022-12-01T10:18:39.4581701Z   test_load_rowwise_to_colwise (__main__.TestDistributedReshardOnLoad) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-12-01T10:18:39.4582137Z   test_load_with_different_shard_plan (__main__.TestDistributedReshardOnLoad) ... skip: c10d was not compiled with the NCCL backend (0.002s)
2022-12-01T10:18:39.4582611Z   test_save_load_bytes (__main__.TestDistributedReshardOnLoad) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-12-01T10:18:39.4583093Z   test_switch_between_sharded_tensor_to_tensor (__main__.TestDistributedReshardOnLoad) ... skip: c10d was not compiled with the NCCL backend (0.002s)
2022-12-01T10:18:39.4583548Z   test_read_write_only_tensor (__main__.TestDistributedStateDictSaveLoad) ... ok (0.256s)
2022-12-01T10:18:39.4583999Z   test_read_write_shard_tensor (__main__.TestDistributedStateDictSaveLoadWithSharedTensor) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-12-01T10:18:39.4584283Z 
2022-12-01T10:18:39.4584554Z ----------------------------------------------------------------------
2022-12-01T10:18:39.4584833Z Ran 6 tests in 0.263s
2022-12-01T10:18:39.4584949Z 
2022-12-01T10:18:39.4585032Z OK (skipped=5)
2022-12-01T10:18:39.4585195Z 
2022-12-01T10:18:39.4585329Z Generating XML reports...
2022-12-01T10:18:39.4585992Z Generated XML report: test-reports/python-unittest/distributed._shard.checkpoint.test_file_system_checkpoint/TEST-TestDistributedStateDictSaveLoad-20221201101838.xml
2022-12-01T10:18:39.4586714Z Generated XML report: test-reports/python-unittest/distributed._shard.checkpoint.test_file_system_checkpoint/TEST-TestDistributedReshardOnLoad-20221201101838.xml
2022-12-01T10:18:39.4587679Z Generated XML report: test-reports/python-unittest/distributed._shard.checkpoint.test_file_system_checkpoint/TEST-TestDistributedStateDictSaveLoadWithSharedTensor-20221201101838.xml
2022-12-01T10:18:39.4588119Z 
2022-12-01T10:18:39.4588362Z ##[endgroup]
2022-12-01T10:18:39.4588891Z FINISHED PRINTING LOG FILE of distributed/_shard/checkpoint/test_file_system_checkpoint (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-checkpoint-test_file_system_checkpoint_3zug31e3)
2022-12-01T10:18:39.4589196Z 
2022-12-01T10:18:39.4589412Z Running distributed/_shard/checkpoint/test_planner ... [2022-12-01 10:18:39.458087]
2022-12-01T10:18:39.4589964Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/checkpoint/test_planner.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-12-01 10:18:39.458405]
2022-12-01T10:18:40.9583030Z 
2022-12-01T10:18:40.9583627Z Expand the folded group to see the log file of distributed/_shard/checkpoint/test_planner
2022-12-01T10:18:40.9584873Z ##[group]PRINTING LOG FILE of distributed/_shard/checkpoint/test_planner (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-checkpoint-test_planner_r36amre3)
2022-12-01T10:18:40.9585548Z Test results will be stored in test-reports/python-unittest/distributed._shard.checkpoint.test_planner
2022-12-01T10:18:40.9585765Z 
2022-12-01T10:18:40.9585840Z Running tests...
2022-12-01T10:18:40.9586135Z ----------------------------------------------------------------------
2022-12-01T10:18:40.9586432Z   test_global_plan (__main__.TestSavePlan) ... ok (0.245s)
2022-12-01T10:18:40.9586710Z   test_load_with_resharding (__main__.TestSavePlan) ... ok (0.004s)
2022-12-01T10:18:40.9586991Z   test_load_with_world_size_diff_by_one (__main__.TestSavePlan) ... ok (0.003s)
2022-12-01T10:18:40.9587276Z   test_local_load_plan (__main__.TestSavePlan) ... ok (0.003s)
2022-12-01T10:18:40.9587576Z   test_local_plan (__main__.TestSavePlan) ... ok (0.003s)
2022-12-01T10:18:40.9587791Z 
2022-12-01T10:18:40.9588059Z ----------------------------------------------------------------------
2022-12-01T10:18:40.9588334Z Ran 5 tests in 0.258s
2022-12-01T10:18:40.9588453Z 
2022-12-01T10:18:40.9588517Z OK
2022-12-01T10:18:40.9588612Z 
2022-12-01T10:18:40.9588696Z Generating XML reports...
2022-12-01T10:18:40.9589149Z Generated XML report: test-reports/python-unittest/distributed._shard.checkpoint.test_planner/TEST-TestSavePlan-20221201101840.xml
2022-12-01T10:18:40.9589410Z 
2022-12-01T10:18:40.9589643Z ##[endgroup]
2022-12-01T10:18:40.9590129Z FINISHED PRINTING LOG FILE of distributed/_shard/checkpoint/test_planner (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-checkpoint-test_planner_r36amre3)
2022-12-01T10:18:40.9590404Z 
2022-12-01T10:18:40.9590632Z Running distributed/fsdp/test_fsdp_param_exec_order_wrap ... [2022-12-01 10:18:40.958668]
2022-12-01T10:18:40.9592237Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_param_exec_order_wrap.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-12-01 10:18:40.959028]
2022-12-01T10:18:48.1427025Z 
2022-12-01T10:18:48.1427770Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_param_exec_order_wrap
2022-12-01T10:18:48.1429136Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_param_exec_order_wrap (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_param_exec_order_wrap_a4an0ycv)
2022-12-01T10:18:48.1430414Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_param_exec_order_wrap
2022-12-01T10:18:48.1430808Z 
2022-12-01T10:18:48.1430911Z Running tests...
2022-12-01T10:18:48.1431226Z ----------------------------------------------------------------------
2022-12-01T10:18:48.1431639Z   test_fsdp_flatten_params_exec_order_sharding_strategy_ShardingStrategy_FULL_SHARD (__main__.TestFSDPExecOrder)
2022-12-01T10:18:48.1432098Z Test ``_fsdp_params_exec_order`` with ``ParamExecOrderWrapPolicy``, ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17766
2022-12-01T10:18:48.1432508Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17767
2022-12-01T10:18:48.1433326Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17768
2022-12-01T10:18:48.1433795Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 17769
2022-12-01T10:18:48.1434275Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:18:48.1434600Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:18:48.1435037Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:18:48.1435390Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:18:48.1435732Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:18:48.1436190Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:18:48.1436530Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:18:48.1436964Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:18:48.1437313Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:18:48.1437635Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:18:48.1438102Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:18:48.1438430Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:18:48.1438845Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:18:48.1439185Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:18:48.1439525Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-12-01T10:18:48.1439991Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:18:48.1440306Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:18:48.1440735Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:18:48.1441077Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:18:48.1441399Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:18:48.1441895Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:18:48.1442412Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:18:48.1442933Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:18:48.1443436Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:18:48.1443823Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:18:48.1444174Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:18:48.1444517Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:18:48.1444846Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:18:48.1445104Z dist init r=2, world=4
2022-12-01T10:18:48.1445285Z dist init r=1, world=4
2022-12-01T10:18:48.1445450Z dist init r=3, world=4
2022-12-01T10:18:48.1445628Z dist init r=0, world=4
2022-12-01T10:18:48.1445838Z skip: Need at least 2 CUDA devices (1.658s)
2022-12-01T10:18:48.1446208Z   test_fsdp_flatten_params_exec_order_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestFSDPExecOrder)
2022-12-01T10:18:48.1446697Z Test ``_fsdp_params_exec_order`` with ``ParamExecOrderWrapPolicy``, ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17822
2022-12-01T10:18:48.1447103Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17823
2022-12-01T10:18:48.1447436Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17824
2022-12-01T10:18:48.1447746Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 17825
2022-12-01T10:18:48.1448203Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:18:48.1448538Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:18:48.1448972Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:18:48.1449315Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:18:48.1449653Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:18:48.1450249Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:18:48.1450565Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:18:48.1450993Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:18:48.1451335Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:18:48.1451667Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:18:48.1452115Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:18:48.1452441Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:18:48.1452870Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:18:48.1453215Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:18:48.1453537Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-12-01T10:18:48.1454031Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:18:48.1454548Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:18:48.1455030Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:18:48.1455359Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:18:48.1455789Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:18:48.1456136Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:18:48.1456458Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:18:48.1456952Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:18:48.1457556Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:18:48.1457945Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:18:48.1458281Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:18:48.1458623Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:18:48.1459028Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:18:48.1459315Z dist init r=0, world=4
2022-12-01T10:18:48.1459501Z dist init r=1, world=4
2022-12-01T10:18:48.1459686Z dist init r=3, world=4
2022-12-01T10:18:48.1459851Z dist init r=2, world=4
2022-12-01T10:18:48.1460063Z skip: Need at least 2 CUDA devices (1.419s)
2022-12-01T10:18:48.1460607Z   test_fsdp_flatten_params_exec_order_symbolic_trace_sharding_strategy_ShardingStrategy_FULL_SHARD (__main__.TestFSDPExecOrder)
2022-12-01T10:18:48.1461074Z Tests ``ParamExecOrderWrapPolicy`` with symbolic tracing. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17878
2022-12-01T10:18:48.1461461Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17879
2022-12-01T10:18:48.1461793Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17880
2022-12-01T10:18:48.1462124Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 17881
2022-12-01T10:18:48.1462578Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:18:48.1462912Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:18:48.1463339Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:18:48.1463685Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:18:48.1464004Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:18:48.1464470Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:18:48.1464799Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:18:48.1465226Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:18:48.1465557Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:18:48.1465894Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:18:48.1466360Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:18:48.1466670Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:18:48.1467094Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:18:48.1467436Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:18:48.1467764Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-12-01T10:18:48.1468243Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:18:48.1468737Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:18:48.1469063Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:18:48.1469490Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:18:48.1469818Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:18:48.1470152Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:18:48.1470644Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:18:48.1471149Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:18:48.1471658Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:18:48.1472117Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:18:48.1472510Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:18:48.1472842Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:18:48.1473201Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:18:48.1473527Z dist init r=3, world=4
2022-12-01T10:18:48.1473696Z dist init r=0, world=4
2022-12-01T10:18:48.1473873Z dist init r=1, world=4
2022-12-01T10:18:48.1474049Z dist init r=2, world=4
2022-12-01T10:18:48.1474245Z skip: Need at least 2 CUDA devices (1.417s)
2022-12-01T10:18:48.1474611Z   test_fsdp_flatten_params_exec_order_symbolic_trace_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestFSDPExecOrder)
2022-12-01T10:18:48.1475400Z Tests ``ParamExecOrderWrapPolicy`` with symbolic tracing. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17934
2022-12-01T10:18:48.1475920Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17935
2022-12-01T10:18:48.1476241Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17936
2022-12-01T10:18:48.1476557Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 17937
2022-12-01T10:18:48.1477022Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:18:48.1477352Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:18:48.1477768Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:18:48.1478112Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:18:48.1478444Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:18:48.1478902Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:18:48.1479232Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:18:48.1479659Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:18:48.1480000Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:18:48.1480319Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:18:48.1480784Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:18:48.1481109Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:18:48.1481532Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:18:48.1481865Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:18:48.1482203Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:18:48.1482668Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:18:48.1482977Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:18:48.1483402Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:18:48.1483748Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:18:48.1484084Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-12-01T10:18:48.1484570Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:18:48.1485146Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:18:48.1485710Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:18:48.1486223Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:18:48.1486599Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:18:48.1486951Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:18:48.1487301Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:18:48.1487630Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:18:48.1487892Z dist init r=3, world=4
2022-12-01T10:18:48.1488078Z dist init r=0, world=4
2022-12-01T10:18:48.1488243Z dist init r=1, world=4
2022-12-01T10:18:48.1488423Z dist init r=2, world=4
2022-12-01T10:18:48.1488633Z skip: Need at least 2 CUDA devices (1.417s)
2022-12-01T10:18:48.1488771Z 
2022-12-01T10:18:48.1488973Z ----------------------------------------------------------------------
2022-12-01T10:18:48.1489207Z Ran 4 tests in 5.911s
2022-12-01T10:18:48.1489321Z 
2022-12-01T10:18:48.1489394Z OK (skipped=4)
2022-12-01T10:18:48.1489501Z 
2022-12-01T10:18:48.1489585Z Generating XML reports...
2022-12-01T10:18:48.1490048Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_param_exec_order_wrap/TEST-TestFSDPExecOrder-20221201101841.xml
2022-12-01T10:18:48.1490323Z 
2022-12-01T10:18:48.1490652Z ##[endgroup]
2022-12-01T10:18:48.1491152Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_param_exec_order_wrap (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_param_exec_order_wrap_a4an0ycv)
2022-12-01T10:18:48.1491442Z 
2022-12-01T10:18:48.1491664Z Running distributed/test_distributed_spawn ... [2022-12-01 10:18:48.143085]
2022-12-01T10:18:48.1491975Z MPI not available -- MPI backend tests will be skipped
2022-12-01T10:18:48.1525232Z Running distributed tests for the test backend with env init_method
2022-12-01T10:18:48.1541814Z Running distributed tests for the gloo backend with env init_method
2022-12-01T10:18:48.1553580Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-12-01 10:18:48.155115]
2022-12-01T10:18:48.1564896Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-12-01 10:18:48.156233]
2022-12-01T10:18:49.5427984Z 
2022-12-01T10:18:49.5428577Z Expand the folded group to see the log file of distributed/test_distributed_spawn
2022-12-01T10:18:49.5429759Z ##[group]PRINTING LOG FILE of distributed/test_distributed_spawn (/var/lib/jenkins/workspace/test/test-reports/distributed-test_distributed_spawn_c1hpfu5s)
2022-12-01T10:18:49.5430427Z <unittest.suite.TestSuite tests=[]>
2022-12-01T10:18:49.5430656Z 
2022-12-01T10:18:49.5431009Z ##[endgroup]
2022-12-01T10:18:49.5431920Z FINISHED PRINTING LOG FILE of distributed/test_distributed_spawn (/var/lib/jenkins/workspace/test/test-reports/distributed-test_distributed_spawn_c1hpfu5s)
2022-12-01T10:18:49.5432378Z 
2022-12-01T10:31:01.9034069Z 
2022-12-01T10:31:01.9034649Z Expand the folded group to see the log file of distributed/test_distributed_spawn
2022-12-01T10:31:01.9035730Z ##[group]PRINTING LOG FILE of distributed/test_distributed_spawn (/var/lib/jenkins/workspace/test/test-reports/distributed-test_distributed_spawn_lt5tdku2)
2022-12-01T10:31:01.9047391Z <unittest.suite.TestSuite tests=[]>
2022-12-01T10:31:01.9114345Z <unittest.suite.TestSuite tests=[<__main__.TestDistBackendWithSpawn testMethod=test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager>, <__main__.TestDistBackendWithSpawn testMethod=test_3_level_hierarchical_model_averager>, <__main__.TestDistBackendWithSpawn testMethod=test_Backend_enum_class>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallelCPU>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallelCPU_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_2D_Input>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Channels_Last>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_No_Affine>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_non_default_stream>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_requires_grad>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_with_amp_and_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedSampler_padding>, <__main__.TestDistBackendWithSpawn testMethod=test_SyncBatchNorm_process_group>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync_allreduce_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync_allreduce_with_then_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_simple>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_with_empty>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_into_cat_tensor_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_into_stack_tensor_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_multigpu_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_object_default_pg>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_object_subgroup>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_v_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_max_complex_unsupported>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_complex_unsupported_ops>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_multigpu_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_result_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_async>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_cuda_async>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_average_parameters>, <__main__.TestDistBackendWithSpawn testMethod=test_backend_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_backend_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_timeout_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_timeout_global>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_timeout_group>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_gloo>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_gloo_tags>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_mixed_backend_err>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_no_rank_zero_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_op_err>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_op_list_err>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_ring_exchange_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_self_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_tensor_err>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_group>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_object_list>, <__main__.TestDistBackendWithSpawn testMethod=test_compute_bucket_assignment_by_size_sparse_error_with_logger>, <__main__.TestDistBackendWithSpawn testMethod=test_compute_bucket_assignment_by_size_sparse_error_without_logger>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_broadcast_buffer>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_broadcast_buffer_via_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_buffer_hook_allreduce>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_buffer_hook_allreduce_return_future>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_build_debug_param_to_name_mapping>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_build_debug_param_to_name_mapping_requires_grad>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_comm_hook_logging>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_control_flow_different_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_control_flow_same_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_create_graph>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_device>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_forward_backward_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_grad_div_uneven_inputs>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_allreduce>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_allreduce_process_group>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_post_localSGD>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_powerSGD>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_pickling_powerSGD>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adam_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adam_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_ignore_params_arg>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_inference>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_join_model_equivalence>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_logging_data_cpu>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_logging_data_gpu>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_model_diff_num_params_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_model_diff_shape_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_multiple_nested_unused_params_err_ignore_params>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_multiple_nested_unused_params_error>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_namedtuple>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_new_tensor_in_fwd>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_new_tensor_in_fwd_static_graph>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_profiling_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_profiling_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_python_error_logged>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_returns_tensor_with_no_grad>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_shared_grad_acc_unused_params>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_static_graph_nested_types>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_sync_bn_training_vs_eval>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_sync_module_states>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_input_exception>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_input_join_disable>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_inputs>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_inputs_stop_iteration_sync_bn>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_unused_params_rebuild_buckets_exception>, <__main__.TestDistBackendWithSpawn testMethod=test_destroy_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_destroy_group>, <__main__.TestDistBackendWithSpawn testMethod=test_detect_ddp_is_actually_static>, <__main__.TestDistBackendWithSpawn testMethod=test_different_graph_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_dump_DDP_relevant_env_vars>, <__main__.TestDistBackendWithSpawn testMethod=test_gather>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_checks>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_group>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_object>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_object_subgroup>, <__main__.TestDistBackendWithSpawn testMethod=test_get_backend>, <__main__.TestDistBackendWithSpawn testMethod=test_get_future>, <__main__.TestDistBackendWithSpawn testMethod=test_get_rank>, <__main__.TestDistBackendWithSpawn testMethod=test_get_rank_size_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_get_rank_size_group>, <__main__.TestDistBackendWithSpawn testMethod=test_invalid_static_graph>, <__main__.TestDistBackendWithSpawn testMethod=test_irecv>, <__main__.TestDistBackendWithSpawn testMethod=test_isend>, <__main__.TestDistBackendWithSpawn testMethod=test_isend_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_isend_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_allreduce_hang>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_allreduce_hang_wait_all_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_failure_order>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_gloo>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_gloo_rank_0_timeout>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_gloo_subgroup>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_wait_all_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_allgather>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_allreduce>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_broadcast>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_reduce>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_high_priority_stream>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_by_enumeration>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_by_enumeration_input_rank_exceeds_world_size>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_by_enumeration_negative_input_rank>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_group_size_exceeds_world_size>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_overlap_not_allowed>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_world_size_not_divisible_by_group_size>, <__main__.TestDistBackendWithSpawn testMethod=test_output_unused_in_loss_dict_module>, <__main__.TestDistBackendWithSpawn testMethod=test_output_unused_in_loss_tuple_module>, <__main__.TestDistBackendWithSpawn testMethod=test_periodic_model_averager>, <__main__.TestDistBackendWithSpawn testMethod=test_periodic_model_averager_param_group>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity_with_hierarchical_sgd>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_step_reload>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_max>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_min>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_product>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_scatter_tensor_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_scatter_v_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum_cuda_twice>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum_twice>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_checks>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_group>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_object_list>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_any_source>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_any_source_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_any_source_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_nccl_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_nccl_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_with_tag>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_with_tag_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_with_tag_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_sparse_all_reduce_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_sparse_all_reduce_sum_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_stateless_api_with_ddp>, <__main__.TestDistBackendWithSpawn testMethod=test_static_graph_api_cpu>, <__main__.TestDistBackendWithSpawn testMethod=test_sync_bn_logged>, <__main__.TestDistBackendWithSpawn testMethod=test_undefined_grad_parity_unused_parameters>, <__main__.TestDistBackendWithSpawn testMethod=test_verify_model_across_rank_with_logger>, <__main__.TestDistBackendWithSpawn testMethod=test_verify_model_across_rank_without_logger>]>
2022-12-01T10:31:01.9162201Z test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9162604Z test_3_level_hierarchical_model_averager (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9163002Z test_Backend_enum_class (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9163457Z test_DistributedDataParallel (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9163960Z test_DistributedDataParallelCPU (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9164455Z test_DistributedDataParallelCPU_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9164906Z test_DistributedDataParallel_SyncBatchNorm (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9165348Z test_DistributedDataParallel_SyncBatchNorm_2D_Input (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9165854Z test_DistributedDataParallel_SyncBatchNorm_Channels_Last (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9166321Z test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9166818Z test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9167340Z test_DistributedDataParallel_SyncBatchNorm_No_Affine (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9168033Z test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9168642Z test_DistributedDataParallel_non_default_stream (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9169148Z test_DistributedDataParallel_requires_grad (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9169617Z test_DistributedDataParallel_with_amp_and_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9170042Z test_DistributedSampler_padding (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9170468Z test_SyncBatchNorm_process_group (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9170870Z test_accumulate_gradients_no_sync (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9171281Z test_accumulate_gradients_no_sync_allreduce_hook (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9171756Z test_accumulate_gradients_no_sync_allreduce_with_then_hook (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9172288Z test_accumulate_gradients_no_sync_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9172710Z test_all_gather (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9173165Z test_all_gather_coalesced_complex (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9173590Z test_all_gather_coalesced_full_group (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9173974Z test_all_gather_coalesced_group (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9174282Z test_all_gather_coalesced_simple (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9174581Z test_all_gather_coalesced_with_empty (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9174904Z test_all_gather_complex (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9175364Z test_all_gather_cuda (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9175702Z test_all_gather_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9175998Z test_all_gather_full_group (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9176290Z test_all_gather_group (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9176714Z test_all_gather_into_cat_tensor_cuda (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9177169Z test_all_gather_into_stack_tensor_cuda (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9177599Z test_all_gather_multigpu (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9178008Z test_all_gather_multigpu_complex (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9178433Z test_all_gather_object_default_pg (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9178923Z test_all_gather_object_subgroup (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9179372Z test_all_gather_v_cuda (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9179724Z test_all_reduce_coalesced_full_group_max (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9180124Z test_all_reduce_coalesced_full_group_min (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9180709Z test_all_reduce_coalesced_full_group_product (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9181240Z test_all_reduce_coalesced_full_group_sum (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9181770Z test_all_reduce_coalesced_group_max (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9182192Z test_all_reduce_coalesced_group_min (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9182641Z test_all_reduce_coalesced_group_product (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9183118Z test_all_reduce_coalesced_group_sum (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9183535Z test_all_reduce_coalesced_max (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9183994Z test_all_reduce_coalesced_max_complex_unsupported (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9184535Z test_all_reduce_coalesced_min (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9184945Z test_all_reduce_coalesced_product (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9185434Z test_all_reduce_coalesced_sum (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9185864Z test_all_reduce_complex_unsupported_ops (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9186455Z test_all_reduce_full_group_max (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9186941Z test_all_reduce_full_group_min (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9187495Z test_all_reduce_full_group_product (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9187942Z test_all_reduce_full_group_sum (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9188391Z test_all_reduce_group_max (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9188833Z test_all_reduce_group_min (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9189288Z test_all_reduce_group_product (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9189749Z test_all_reduce_group_sum (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9190196Z test_all_reduce_max (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9190603Z test_all_reduce_min (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9191015Z test_all_reduce_multigpu (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9191456Z test_all_reduce_multigpu_complex (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9191859Z test_all_reduce_product (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9192317Z test_all_reduce_result_cuda (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9192720Z test_all_reduce_sum (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9193022Z test_all_reduce_sum_async (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9193431Z test_all_reduce_sum_complex (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9193734Z test_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9194013Z test_all_reduce_sum_cuda_async (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9194320Z test_all_reduce_sum_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9194610Z test_all_to_all (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9194873Z test_all_to_all_complex (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9195157Z test_all_to_all_cuda (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9195444Z test_all_to_all_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9195744Z test_all_to_all_full_group (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9196027Z test_all_to_all_full_group_cuda (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9196320Z test_all_to_all_group (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9196607Z test_all_to_all_group_cuda (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9196893Z test_all_to_all_single_equal_split (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9197213Z test_all_to_all_single_equal_split_complex (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9197535Z test_all_to_all_single_equal_split_cuda (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9197864Z test_all_to_all_single_equal_split_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9198188Z test_all_to_all_single_equal_split_full_group (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9198523Z test_all_to_all_single_equal_split_full_group_cuda (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9198854Z test_all_to_all_single_equal_split_group (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9199171Z test_all_to_all_single_equal_split_group_cuda (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9199489Z test_all_to_all_single_unequal_split (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9199810Z test_all_to_all_single_unequal_split_complex (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9200125Z test_all_to_all_single_unequal_split_cuda (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9200458Z test_all_to_all_single_unequal_split_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9200849Z test_all_to_all_single_unequal_split_full_group (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9201257Z test_all_to_all_single_unequal_split_full_group_cuda (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9201618Z test_all_to_all_single_unequal_split_group (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9202018Z test_all_to_all_single_unequal_split_group_cuda (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9202338Z test_average_parameters (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9202618Z test_backend_full_group (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9202899Z test_backend_group (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9203167Z test_barrier (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9203435Z test_barrier_cuda (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9203701Z test_barrier_full_group (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9203991Z test_barrier_full_group_cuda (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9213472Z test_barrier_group (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9213866Z test_barrier_group_cuda (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9214164Z test_barrier_timeout_full_group (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9214468Z test_barrier_timeout_global (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9214767Z test_barrier_timeout_group (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9215049Z test_batch_isend_irecv_gloo (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9215426Z test_batch_isend_irecv_gloo_tags (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9215747Z test_batch_isend_irecv_mixed_backend_err (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9216046Z test_batch_isend_irecv_nccl (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9216356Z test_batch_isend_irecv_no_rank_zero_nccl (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9216661Z test_batch_isend_irecv_op_err (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9216968Z test_batch_isend_irecv_op_list_err (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9217278Z test_batch_isend_irecv_ring_exchange_nccl (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9217596Z test_batch_isend_irecv_self_nccl (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9217901Z test_batch_isend_irecv_tensor_err (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9218174Z test_broadcast (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9218450Z test_broadcast_cuda (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9218740Z test_broadcast_full_group (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9219014Z test_broadcast_group (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9219302Z test_broadcast_multigpu (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9219594Z test_broadcast_object_list (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9219931Z test_compute_bucket_assignment_by_size_sparse_error_with_logger (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9220303Z test_compute_bucket_assignment_by_size_sparse_error_without_logger (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9220866Z test_ddp_broadcast_buffer (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9221241Z test_ddp_broadcast_buffer_via_hook (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9221737Z test_ddp_buffer_hook_allreduce (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9222260Z test_ddp_buffer_hook_allreduce_return_future (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9222856Z test_ddp_build_debug_param_to_name_mapping (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9223428Z test_ddp_build_debug_param_to_name_mapping_requires_grad (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9223940Z test_ddp_comm_hook_logging (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9224531Z test_ddp_control_flow_different_across_ranks (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9225241Z test_ddp_control_flow_same_across_ranks (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9225847Z test_ddp_create_graph (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9226286Z test_ddp_device (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9226836Z test_ddp_forward_backward_hook (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9227334Z test_ddp_grad_div_uneven_inputs (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9227869Z test_ddp_hook_parity_allreduce (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9228412Z test_ddp_hook_parity_allreduce_process_group (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9228934Z test_ddp_hook_parity_post_localSGD (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9229495Z test_ddp_hook_parity_powerSGD (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9230043Z test_ddp_hook_pickling_powerSGD (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9230640Z test_ddp_hook_with_optimizer_parity_adam_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9231065Z test_ddp_hook_with_optimizer_parity_adam_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9231480Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9231940Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9232399Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9232857Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9233361Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9233798Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9234247Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9234697Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9235104Z test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9235458Z test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9235785Z test_ddp_ignore_params_arg (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9236110Z test_ddp_inference (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9236393Z test_ddp_join_model_equivalence (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9236699Z test_ddp_logging_data_cpu (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9236995Z test_ddp_logging_data_gpu (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9237308Z test_ddp_model_diff_num_params_across_ranks (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9237617Z test_ddp_model_diff_shape_across_ranks (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9237961Z test_ddp_multiple_nested_unused_params_err_ignore_params (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9238305Z test_ddp_multiple_nested_unused_params_error (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9238599Z test_ddp_namedtuple (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9238886Z test_ddp_new_tensor_in_fwd (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9239193Z test_ddp_new_tensor_in_fwd_static_graph (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9239515Z test_ddp_profiling_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9239878Z test_ddp_profiling_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9240221Z test_ddp_python_error_logged (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9240528Z test_ddp_returns_tensor_with_no_grad (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9240833Z test_ddp_shared_grad_acc_unused_params (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9241155Z test_ddp_static_graph_nested_types (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9241468Z test_ddp_sync_bn_training_vs_eval (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9241801Z test_ddp_sync_module_states (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9242103Z test_ddp_uneven_input_exception (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9242415Z test_ddp_uneven_input_join_disable (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9242715Z test_ddp_uneven_inputs (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9243016Z test_ddp_uneven_inputs_stop_iteration_sync_bn (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9243359Z test_ddp_unused_params_rebuild_buckets_exception (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9243677Z test_destroy_full_group (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9243944Z test_destroy_group (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9244238Z test_detect_ddp_is_actually_static (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9244551Z test_different_graph_across_ranks (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9244858Z test_dump_DDP_relevant_env_vars (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9245128Z test_gather (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9245398Z test_gather_checks (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9245670Z test_gather_cuda (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9245939Z test_gather_full_group (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9246219Z test_gather_group (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9246492Z test_gather_object (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9246769Z test_gather_object_subgroup (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9247053Z test_get_backend (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9247323Z test_get_future (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9247574Z test_get_rank (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9247855Z test_get_rank_size_full_group (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9248148Z test_get_rank_size_group (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9248436Z test_invalid_static_graph (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9248694Z test_irecv (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9248953Z test_isend (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9249232Z test_isend_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9249518Z test_isend_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9249823Z test_monitored_barrier_allreduce_hang (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9250164Z test_monitored_barrier_allreduce_hang_wait_all_ranks (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9250485Z test_monitored_barrier_failure_order (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9250793Z test_monitored_barrier_gloo (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9251106Z test_monitored_barrier_gloo_rank_0_timeout (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9251427Z test_monitored_barrier_gloo_subgroup (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9251731Z test_monitored_barrier_wait_all_ranks (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9252045Z test_nccl_backend_bool_allgather (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9252355Z test_nccl_backend_bool_allreduce (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9252711Z test_nccl_backend_bool_broadcast (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9253093Z test_nccl_backend_bool_reduce (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9253450Z test_nccl_high_priority_stream (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9253742Z test_new_subgroups (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9254080Z test_new_subgroups_by_enumeration (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9254428Z test_new_subgroups_by_enumeration_input_rank_exceeds_world_size (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9254791Z test_new_subgroups_by_enumeration_negative_input_rank (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9255126Z test_new_subgroups_group_size_exceeds_world_size (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9255541Z test_new_subgroups_overlap_not_allowed (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9255883Z test_new_subgroups_world_size_not_divisible_by_group_size (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9256225Z test_output_unused_in_loss_dict_module (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9256532Z test_output_unused_in_loss_tuple_module (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9256840Z test_periodic_model_averager (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9257158Z test_periodic_model_averager_param_group (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9257470Z test_post_localSGD_optimizer_parity (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9257802Z test_post_localSGD_optimizer_parity_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9258158Z test_post_localSGD_optimizer_parity_with_hierarchical_sgd (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9258536Z test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9258877Z test_post_localSGD_optimizer_step_reload (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9259192Z test_reduce_full_group_max (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9259483Z test_reduce_full_group_min (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9259775Z test_reduce_full_group_product (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9260072Z test_reduce_full_group_sum (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9260521Z test_reduce_group_max (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9260795Z test_reduce_group_min (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9261092Z test_reduce_group_product (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9261377Z test_reduce_group_sum (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9261648Z test_reduce_max (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9261904Z test_reduce_min (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9262181Z test_reduce_multigpu (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9262464Z test_reduce_product (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9262826Z test_reduce_scatter_tensor_cuda (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9263205Z test_reduce_scatter_v_cuda (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9263526Z test_reduce_sum (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9263911Z test_reduce_sum_cuda (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9264230Z test_reduce_sum_cuda_twice (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9264736Z test_reduce_sum_twice (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9265045Z test_scatter (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9265381Z test_scatter_checks (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9265695Z test_scatter_complex (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9266067Z test_scatter_cuda (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9266396Z test_scatter_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9266748Z test_scatter_full_group (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9267139Z test_scatter_group (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9267525Z test_scatter_object_list (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9267838Z test_send_recv (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9268156Z test_send_recv_any_source (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9268562Z test_send_recv_any_source_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9268939Z test_send_recv_any_source_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9269321Z test_send_recv_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9269658Z test_send_recv_nccl (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9270040Z test_send_recv_nccl_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9270382Z test_send_recv_nccl_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9270766Z test_send_recv_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9271120Z test_send_recv_with_tag (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9271518Z test_send_recv_with_tag_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9271879Z test_send_recv_with_tag_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9272266Z test_sparse_all_reduce_sum (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9272642Z test_sparse_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9273054Z test_stateless_api_with_ddp (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9273483Z test_static_graph_api_cpu (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9273922Z test_sync_bn_logged (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9274269Z test_undefined_grad_parity_unused_parameters (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9274718Z test_verify_model_across_rank_with_logger (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9275087Z test_verify_model_across_rank_without_logger (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9275693Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:01.9276270Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:01.9276655Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:01.9277202Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:01.9277595Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:01.9277784Z 
2022-12-01T10:31:01.9277866Z Running tests...
2022-12-01T10:31:01.9278206Z ----------------------------------------------------------------------
2022-12-01T10:31:01.9278794Z   test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18027
2022-12-01T10:31:01.9279419Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18028
2022-12-01T10:31:01.9279778Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18029
2022-12-01T10:31:01.9280368Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:01.9280819Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:01.9281297Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:01.9281735Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:01.9282106Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:31:01.9282766Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:01.9283189Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:01.9283706Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:01.9284151Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:01.9284579Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:31:01.9285088Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:01.9285511Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:01.9285979Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:01.9286393Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:01.9286797Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:31:01.9287394Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:01.9287999Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:01.9288635Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:01.9289109Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:31:01.9289565Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:31:01.9289996Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:31:01.9290294Z skip: Need at least 2 CUDA devices (1.855s)
2022-12-01T10:31:01.9290438Z 
2022-12-01T10:31:01.9290688Z ----------------------------------------------------------------------
2022-12-01T10:31:01.9290968Z Ran 1 test in 1.855s
2022-12-01T10:31:01.9291087Z 
2022-12-01T10:31:01.9291187Z OK (skipped=1)
2022-12-01T10:31:01.9291283Z 
2022-12-01T10:31:01.9291377Z Generating XML reports...
2022-12-01T10:31:01.9292051Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201101850.xml
2022-12-01T10:31:01.9292735Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:01.9293287Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:01.9293663Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:01.9294188Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:01.9294622Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:01.9294801Z 
2022-12-01T10:31:01.9294884Z Running tests...
2022-12-01T10:31:01.9295314Z ----------------------------------------------------------------------
2022-12-01T10:31:01.9295777Z   test_3_level_hierarchical_model_averager (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.003s)
2022-12-01T10:31:01.9296031Z 
2022-12-01T10:31:01.9296292Z ----------------------------------------------------------------------
2022-12-01T10:31:01.9296556Z Ran 1 test in 0.003s
2022-12-01T10:31:01.9296668Z 
2022-12-01T10:31:01.9296755Z OK (skipped=1)
2022-12-01T10:31:01.9296925Z 
2022-12-01T10:31:01.9297020Z Generating XML reports...
2022-12-01T10:31:01.9297523Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201101853.xml
2022-12-01T10:31:01.9298130Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:01.9298761Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:01.9299200Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:01.9299703Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:01.9300165Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:01.9300475Z 
2022-12-01T10:31:01.9300584Z Running tests...
2022-12-01T10:31:01.9300998Z ----------------------------------------------------------------------
2022-12-01T10:31:01.9301485Z   test_Backend_enum_class (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18090
2022-12-01T10:31:01.9301962Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18091
2022-12-01T10:31:01.9302388Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18092
2022-12-01T10:31:01.9302971Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:01.9303369Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:01.9303845Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:01.9304276Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:01.9304666Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:31:01.9305264Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:01.9305671Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:01.9306141Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:01.9306567Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:01.9306960Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:31:01.9307512Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:01.9307929Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:01.9308389Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:01.9308820Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:01.9309200Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:31:01.9309818Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:01.9310419Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:01.9311279Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:01.9311835Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:31:01.9312273Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:31:01.9312680Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:31:01.9312933Z ok (1.752s)
2022-12-01T10:31:01.9313055Z 
2022-12-01T10:31:01.9313291Z ----------------------------------------------------------------------
2022-12-01T10:31:01.9313519Z Ran 1 test in 1.753s
2022-12-01T10:31:01.9313632Z 
2022-12-01T10:31:01.9313693Z OK
2022-12-01T10:31:01.9313798Z 
2022-12-01T10:31:01.9313917Z Generating XML reports...
2022-12-01T10:31:01.9314541Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201101855.xml
2022-12-01T10:31:01.9315132Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:01.9315678Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:01.9316054Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:01.9316505Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:01.9316896Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:01.9317071Z 
2022-12-01T10:31:01.9317171Z Running tests...
2022-12-01T10:31:01.9317511Z ----------------------------------------------------------------------
2022-12-01T10:31:01.9318570Z   test_DistributedDataParallel (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77317 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.237s)
2022-12-01T10:31:01.9319127Z 
2022-12-01T10:31:01.9319375Z ----------------------------------------------------------------------
2022-12-01T10:31:01.9319637Z Ran 1 test in 0.237s
2022-12-01T10:31:01.9319757Z 
2022-12-01T10:31:01.9319841Z OK (skipped=1)
2022-12-01T10:31:01.9320007Z 
2022-12-01T10:31:01.9320108Z Generating XML reports...
2022-12-01T10:31:01.9320599Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201101858.xml
2022-12-01T10:31:01.9321206Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:01.9321771Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:01.9322212Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:01.9322688Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:01.9323154Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:01.9323325Z 
2022-12-01T10:31:01.9323427Z Running tests...
2022-12-01T10:31:01.9323764Z ----------------------------------------------------------------------
2022-12-01T10:31:01.9324262Z   test_DistributedDataParallelCPU (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18153
2022-12-01T10:31:01.9324762Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18154
2022-12-01T10:31:01.9325136Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18155
2022-12-01T10:31:01.9325660Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:01.9326086Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:01.9326613Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:01.9327035Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:01.9327370Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:31:01.9327988Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:01.9328389Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:01.9328879Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:01.9329285Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:01.9329715Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:31:01.9330337Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:01.9330745Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:01.9331256Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:01.9331708Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:01.9332165Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:31:01.9332770Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:01.9333458Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:01.9334110Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:01.9334562Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:31:01.9335015Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:31:01.9335529Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:31:01.9335955Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprhuql8bd
2022-12-01T10:31:01.9336455Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3m4gg79d
2022-12-01T10:31:01.9336870Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvlbl7c4x
2022-12-01T10:31:01.9337392Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprhuql8bd/_remote_module_non_scriptable.py
2022-12-01T10:31:01.9337902Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3m4gg79d/_remote_module_non_scriptable.py
2022-12-01T10:31:01.9338398Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvlbl7c4x/_remote_module_non_scriptable.py
2022-12-01T10:31:01.9338872Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:31:01.9339265Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:31:01.9339647Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:31:01.9339889Z ok (1.855s)
2022-12-01T10:31:01.9339994Z 
2022-12-01T10:31:01.9340208Z ----------------------------------------------------------------------
2022-12-01T10:31:01.9340593Z Ran 1 test in 1.856s
2022-12-01T10:31:01.9340717Z 
2022-12-01T10:31:01.9340779Z OK
2022-12-01T10:31:01.9340859Z 
2022-12-01T10:31:01.9340947Z Generating XML reports...
2022-12-01T10:31:01.9341422Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201101859.xml
2022-12-01T10:31:01.9341929Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:01.9342386Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:01.9342722Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:01.9343240Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:01.9343594Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:01.9343796Z 
2022-12-01T10:31:01.9343859Z Running tests...
2022-12-01T10:31:01.9344204Z ----------------------------------------------------------------------
2022-12-01T10:31:01.9344727Z   test_DistributedDataParallelCPU_grad_is_view (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18206
2022-12-01T10:31:01.9345208Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18207
2022-12-01T10:31:01.9345528Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18208
2022-12-01T10:31:01.9345979Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:01.9346311Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:01.9346729Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:01.9347077Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:01.9347418Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:31:01.9347889Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:01.9348203Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:01.9348633Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:01.9348976Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:01.9349310Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:31:01.9349766Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:01.9350096Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:01.9350521Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:01.9350847Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:01.9351182Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:31:01.9351676Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:01.9352196Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:01.9352698Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:01.9353083Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:31:01.9353450Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_ywm8pyp
2022-12-01T10:31:01.9353853Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_ywm8pyp/_remote_module_non_scriptable.py
2022-12-01T10:31:01.9354217Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:31:01.9354569Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:31:01.9354938Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpss2mh9tx
2022-12-01T10:31:01.9355315Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpotxxmjr3
2022-12-01T10:31:01.9355708Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpss2mh9tx/_remote_module_non_scriptable.py
2022-12-01T10:31:01.9356112Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpotxxmjr3/_remote_module_non_scriptable.py
2022-12-01T10:31:01.9356492Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:31:01.9356838Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:31:01.9357193Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:31:01.9357449Z ok (2.052s)
2022-12-01T10:31:01.9357610Z 
2022-12-01T10:31:01.9357802Z ----------------------------------------------------------------------
2022-12-01T10:31:01.9358077Z Ran 1 test in 2.052s
2022-12-01T10:31:01.9358190Z 
2022-12-01T10:31:01.9358252Z OK
2022-12-01T10:31:01.9358345Z 
2022-12-01T10:31:01.9358429Z Generating XML reports...
2022-12-01T10:31:01.9358877Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201101903.xml
2022-12-01T10:31:01.9359368Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:01.9359839Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:01.9360156Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:01.9360586Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:01.9360934Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:01.9361100Z 
2022-12-01T10:31:01.9361172Z Running tests...
2022-12-01T10:31:01.9361464Z ----------------------------------------------------------------------
2022-12-01T10:31:01.9361897Z   test_DistributedDataParallel_SyncBatchNorm (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18259
2022-12-01T10:31:01.9362329Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18260
2022-12-01T10:31:01.9362661Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18261
2022-12-01T10:31:01.9363100Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:01.9363431Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:01.9363861Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:01.9364194Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:01.9364533Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:31:01.9365008Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:01.9365333Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:01.9365751Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:01.9366099Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:01.9366437Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:31:01.9366902Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:01.9367216Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:01.9367641Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:01.9367983Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:01.9368303Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:31:01.9368795Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:01.9369314Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:01.9369832Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:01.9370205Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:31:01.9370589Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:31:01.9370966Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:31:01.9371248Z skip: CUDA is not available. (1.855s)
2022-12-01T10:31:01.9371366Z 
2022-12-01T10:31:01.9371567Z ----------------------------------------------------------------------
2022-12-01T10:31:01.9371812Z Ran 1 test in 1.855s
2022-12-01T10:31:01.9371926Z 
2022-12-01T10:31:01.9371998Z OK (skipped=1)
2022-12-01T10:31:01.9372109Z 
2022-12-01T10:31:01.9372182Z Generating XML reports...
2022-12-01T10:31:01.9372645Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201101906.xml
2022-12-01T10:31:01.9373138Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:01.9373603Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:01.9373923Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:01.9374354Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:01.9374698Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:01.9374860Z 
2022-12-01T10:31:01.9374918Z Running tests...
2022-12-01T10:31:01.9375216Z ----------------------------------------------------------------------
2022-12-01T10:31:01.9375757Z   test_DistributedDataParallel_SyncBatchNorm_2D_Input (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18312
2022-12-01T10:31:01.9376193Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18313
2022-12-01T10:31:01.9376510Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18314
2022-12-01T10:31:01.9376964Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:01.9377298Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:01.9377710Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:01.9378056Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:01.9378395Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:31:01.9378867Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:01.9379182Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:01.9379612Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:01.9379959Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:01.9380296Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:31:01.9380932Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:01.9381269Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:01.9381699Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:01.9388469Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:01.9388827Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:31:01.9389364Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:01.9389992Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:01.9390545Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:01.9390934Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:31:01.9391284Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:31:01.9391632Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:31:01.9391899Z skip: CUDA is not available. (1.856s)
2022-12-01T10:31:01.9392033Z 
2022-12-01T10:31:01.9392234Z ----------------------------------------------------------------------
2022-12-01T10:31:01.9392477Z Ran 1 test in 1.856s
2022-12-01T10:31:01.9392592Z 
2022-12-01T10:31:01.9392651Z OK (skipped=1)
2022-12-01T10:31:01.9392758Z 
2022-12-01T10:31:01.9392845Z Generating XML reports...
2022-12-01T10:31:01.9393307Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201101910.xml
2022-12-01T10:31:01.9393807Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:01.9394257Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:01.9394588Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:01.9395020Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:01.9395368Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:01.9395519Z 
2022-12-01T10:31:01.9395592Z Running tests...
2022-12-01T10:31:01.9395894Z ----------------------------------------------------------------------
2022-12-01T10:31:01.9396344Z   test_DistributedDataParallel_SyncBatchNorm_Channels_Last (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18365
2022-12-01T10:31:01.9396780Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18366
2022-12-01T10:31:01.9397110Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18367
2022-12-01T10:31:01.9397560Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:01.9397888Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:01.9398304Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:01.9398651Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:01.9398987Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:31:01.9399443Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:01.9399773Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:01.9400200Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:01.9400543Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:01.9400864Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:31:01.9401330Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:01.9401660Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:01.9402088Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:01.9402419Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:01.9402802Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:31:01.9403331Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:01.9403831Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:01.9404349Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:01.9404736Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:31:01.9405086Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:31:01.9405417Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:31:01.9405692Z skip: CUDA is not available. (1.755s)
2022-12-01T10:31:01.9405825Z 
2022-12-01T10:31:01.9406025Z ----------------------------------------------------------------------
2022-12-01T10:31:01.9406270Z Ran 1 test in 1.755s
2022-12-01T10:31:01.9406371Z 
2022-12-01T10:31:01.9406445Z OK (skipped=1)
2022-12-01T10:31:01.9406555Z 
2022-12-01T10:31:01.9406640Z Generating XML reports...
2022-12-01T10:31:01.9407102Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201101913.xml
2022-12-01T10:31:01.9407584Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:01.9408048Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:01.9408382Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:01.9408813Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:01.9409150Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:01.9409317Z 
2022-12-01T10:31:01.9409392Z Running tests...
2022-12-01T10:31:01.9409697Z ----------------------------------------------------------------------
2022-12-01T10:31:01.9410143Z   test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18418
2022-12-01T10:31:01.9410602Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18419
2022-12-01T10:31:01.9410932Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18420
2022-12-01T10:31:01.9411385Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:01.9411703Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:01.9412129Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:01.9412476Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:01.9412820Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:31:01.9413275Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:01.9413604Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:01.9414026Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:01.9414356Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:01.9414692Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:31:01.9415158Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:01.9415639Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:01.9416086Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:01.9416431Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:01.9416768Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:31:01.9417252Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:01.9417771Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:01.9418287Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:01.9418673Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:31:01.9419008Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:31:01.9419356Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:31:01.9419633Z skip: CUDA is not available. (1.757s)
2022-12-01T10:31:01.9419765Z 
2022-12-01T10:31:01.9419963Z ----------------------------------------------------------------------
2022-12-01T10:31:01.9420192Z Ran 1 test in 1.757s
2022-12-01T10:31:01.9420305Z 
2022-12-01T10:31:01.9420573Z OK (skipped=1)
2022-12-01T10:31:01.9420694Z 
2022-12-01T10:31:01.9420780Z Generating XML reports...
2022-12-01T10:31:01.9421233Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201101916.xml
2022-12-01T10:31:01.9421727Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:01.9422196Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:01.9422530Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:01.9422945Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:01.9423294Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:01.9423460Z 
2022-12-01T10:31:01.9423533Z Running tests...
2022-12-01T10:31:01.9423822Z ----------------------------------------------------------------------
2022-12-01T10:31:01.9424283Z   test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18471
2022-12-01T10:31:01.9424740Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18472
2022-12-01T10:31:01.9425071Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18473
2022-12-01T10:31:01.9425512Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:01.9425848Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:01.9426279Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:01.9426624Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:01.9426948Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:31:01.9427417Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:01.9427744Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:01.9428157Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:01.9428570Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:01.9428939Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:31:01.9429405Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:01.9429718Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:01.9430141Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:01.9430484Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:01.9430817Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:31:01.9431298Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:01.9431822Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:01.9432447Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:01.9432819Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:31:01.9433170Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:31:01.9433516Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:31:01.9433797Z skip: CUDA is not available. (1.853s)
2022-12-01T10:31:01.9433931Z 
2022-12-01T10:31:01.9434116Z ----------------------------------------------------------------------
2022-12-01T10:31:01.9434359Z Ran 1 test in 1.854s
2022-12-01T10:31:01.9434476Z 
2022-12-01T10:31:01.9434549Z OK (skipped=1)
2022-12-01T10:31:01.9434655Z 
2022-12-01T10:31:01.9434726Z Generating XML reports...
2022-12-01T10:31:01.9435191Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201101919.xml
2022-12-01T10:31:01.9435689Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:01.9436156Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:01.9436472Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:01.9436903Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:01.9437254Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:01.9437418Z 
2022-12-01T10:31:01.9437492Z Running tests...
2022-12-01T10:31:01.9437784Z ----------------------------------------------------------------------
2022-12-01T10:31:01.9438228Z   test_DistributedDataParallel_SyncBatchNorm_No_Affine (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18524
2022-12-01T10:31:01.9438676Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18525
2022-12-01T10:31:01.9438992Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18526
2022-12-01T10:31:01.9439444Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:01.9439778Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:01.9440205Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:01.9440537Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:01.9440875Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:31:01.9441392Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:01.9441737Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:01.9442162Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:01.9442507Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:01.9442897Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:31:01.9443551Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:01.9443880Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:01.9444309Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:01.9444654Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:01.9444977Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:31:01.9445478Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:01.9445998Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:01.9446502Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:01.9446884Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:31:01.9447236Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:31:01.9447582Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:31:01.9447851Z skip: CUDA is not available. (1.859s)
2022-12-01T10:31:01.9447987Z 
2022-12-01T10:31:01.9448187Z ----------------------------------------------------------------------
2022-12-01T10:31:01.9448431Z Ran 1 test in 1.859s
2022-12-01T10:31:01.9448545Z 
2022-12-01T10:31:01.9448618Z OK (skipped=1)
2022-12-01T10:31:01.9448714Z 
2022-12-01T10:31:01.9448807Z Generating XML reports...
2022-12-01T10:31:01.9449269Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201101923.xml
2022-12-01T10:31:01.9449768Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:01.9450219Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:01.9450554Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:01.9450986Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:01.9451338Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:01.9451492Z 
2022-12-01T10:31:01.9451566Z Running tests...
2022-12-01T10:31:01.9451867Z ----------------------------------------------------------------------
2022-12-01T10:31:01.9452327Z   test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18577
2022-12-01T10:31:01.9452774Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18578
2022-12-01T10:31:01.9453106Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18579
2022-12-01T10:31:01.9453551Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:01.9453883Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:01.9454522Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:01.9454874Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:01.9455319Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:31:01.9455799Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:01.9456114Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:01.9456545Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:01.9456892Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:01.9457215Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:31:01.9457681Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:01.9458016Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:01.9458451Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:01.9458779Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:01.9459115Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:31:01.9459610Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:01.9460131Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:01.9460830Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:01.9461221Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:31:01.9461572Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:31:01.9461905Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:31:01.9462244Z skip: CUDA is not available. (1.851s)
2022-12-01T10:31:01.9462402Z 
2022-12-01T10:31:01.9462689Z ----------------------------------------------------------------------
2022-12-01T10:31:01.9463017Z Ran 1 test in 1.851s
2022-12-01T10:31:01.9463119Z 
2022-12-01T10:31:01.9463194Z OK (skipped=1)
2022-12-01T10:31:01.9463301Z 
2022-12-01T10:31:01.9463387Z Generating XML reports...
2022-12-01T10:31:01.9463851Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201101926.xml
2022-12-01T10:31:01.9464331Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:01.9464799Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:01.9465134Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:01.9465561Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:01.9465897Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:01.9466062Z 
2022-12-01T10:31:01.9466136Z Running tests...
2022-12-01T10:31:01.9466437Z ----------------------------------------------------------------------
2022-12-01T10:31:01.9467346Z   test_DistributedDataParallel_non_default_stream (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/76428 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.237s)
2022-12-01T10:31:01.9467926Z 
2022-12-01T10:31:01.9468113Z ----------------------------------------------------------------------
2022-12-01T10:31:01.9468398Z Ran 1 test in 0.237s
2022-12-01T10:31:01.9468512Z 
2022-12-01T10:31:01.9468587Z OK (skipped=1)
2022-12-01T10:31:01.9468696Z 
2022-12-01T10:31:01.9468782Z Generating XML reports...
2022-12-01T10:31:01.9469230Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201101929.xml
2022-12-01T10:31:01.9469729Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:01.9470196Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:01.9470515Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:01.9470944Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:01.9471296Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:01.9471462Z 
2022-12-01T10:31:01.9471536Z Running tests...
2022-12-01T10:31:01.9471824Z ----------------------------------------------------------------------
2022-12-01T10:31:01.9472272Z   test_DistributedDataParallel_requires_grad (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18640
2022-12-01T10:31:01.9472701Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18641
2022-12-01T10:31:01.9473035Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18642
2022-12-01T10:31:01.9473475Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:01.9473808Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:01.9474240Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:01.9474574Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:01.9474954Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:31:01.9475468Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:01.9475799Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:01.9476256Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:01.9476626Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:01.9477027Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:31:01.9477575Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:01.9477908Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:01.9478340Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:01.9478689Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:01.9479011Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:31:01.9479508Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:01.9480033Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:01.9480548Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:01.9480962Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:31:01.9481313Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:31:01.9481693Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:31:01.9481934Z ok (1.955s)
2022-12-01T10:31:01.9482038Z 
2022-12-01T10:31:01.9482239Z ----------------------------------------------------------------------
2022-12-01T10:31:01.9482482Z Ran 1 test in 1.955s
2022-12-01T10:31:01.9482596Z 
2022-12-01T10:31:01.9482657Z OK
2022-12-01T10:31:01.9482736Z 
2022-12-01T10:31:01.9482821Z Generating XML reports...
2022-12-01T10:31:01.9483284Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201101931.xml
2022-12-01T10:31:01.9483783Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:01.9484235Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:01.9484569Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:01.9485002Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:01.9485349Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:01.9485532Z 
2022-12-01T10:31:01.9485617Z Running tests...
2022-12-01T10:31:01.9486007Z ----------------------------------------------------------------------
2022-12-01T10:31:01.9487141Z   test_DistributedDataParallel_with_amp_and_grad_is_view (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77294 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.239s)
2022-12-01T10:31:01.9487635Z 
2022-12-01T10:31:01.9487886Z ----------------------------------------------------------------------
2022-12-01T10:31:01.9488120Z Ran 1 test in 0.239s
2022-12-01T10:31:01.9488234Z 
2022-12-01T10:31:01.9488306Z OK (skipped=1)
2022-12-01T10:31:01.9488414Z 
2022-12-01T10:31:01.9488548Z Generating XML reports...
2022-12-01T10:31:01.9489018Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201101934.xml
2022-12-01T10:31:01.9489554Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:01.9490066Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:01.9490407Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:01.9490879Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:01.9491230Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:01.9491396Z 
2022-12-01T10:31:01.9491477Z Running tests...
2022-12-01T10:31:01.9491828Z ----------------------------------------------------------------------
2022-12-01T10:31:01.9492232Z   test_DistributedSampler_padding (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18703
2022-12-01T10:31:01.9492691Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18704
2022-12-01T10:31:01.9493043Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18705
2022-12-01T10:31:01.9493572Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:01.9493940Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:01.9494450Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:01.9494946Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:01.9495425Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:31:01.9495909Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:01.9496284Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:01.9496714Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:01.9497103Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:01.9497443Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:31:01.9497964Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:01.9498282Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:01.9498772Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:01.9499143Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:01.9499513Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:31:01.9500046Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:01.9500736Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:01.9501260Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:01.9501645Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:31:01.9502047Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:31:01.9502391Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:31:01.9502680Z skip: Need at least 3 CUDA devices (1.752s)
2022-12-01T10:31:01.9502821Z 
2022-12-01T10:31:01.9503010Z ----------------------------------------------------------------------
2022-12-01T10:31:01.9503250Z Ran 1 test in 1.753s
2022-12-01T10:31:01.9503367Z 
2022-12-01T10:31:01.9503440Z OK (skipped=1)
2022-12-01T10:31:01.9503549Z 
2022-12-01T10:31:01.9503634Z Generating XML reports...
2022-12-01T10:31:01.9504123Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201101936.xml
2022-12-01T10:31:01.9504633Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:01.9505098Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:01.9505418Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:01.9505851Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:01.9506208Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:01.9506370Z 
2022-12-01T10:31:01.9506445Z Running tests...
2022-12-01T10:31:01.9506733Z ----------------------------------------------------------------------
2022-12-01T10:31:01.9507087Z   test_SyncBatchNorm_process_group (__main__.TestDistBackendWithSpawn) ... skip: no torchvision (0.002s)
2022-12-01T10:31:01.9507293Z 
2022-12-01T10:31:01.9507488Z ----------------------------------------------------------------------
2022-12-01T10:31:01.9507726Z Ran 1 test in 0.002s
2022-12-01T10:31:01.9507849Z 
2022-12-01T10:31:01.9507954Z OK (skipped=1)
2022-12-01T10:31:01.9508066Z 
2022-12-01T10:31:01.9508153Z Generating XML reports...
2022-12-01T10:31:01.9508681Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201101939.xml
2022-12-01T10:31:01.9509201Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:01.9509661Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:01.9509994Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:01.9510427Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:01.9510763Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:01.9510928Z 
2022-12-01T10:31:01.9511001Z Running tests...
2022-12-01T10:31:01.9511302Z ----------------------------------------------------------------------
2022-12-01T10:31:01.9511606Z   test_accumulate_gradients_no_sync (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9512009Z Runs _test_accumulate_gradients_no_sync using default inputs ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18766
2022-12-01T10:31:01.9512408Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18767
2022-12-01T10:31:01.9512739Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18768
2022-12-01T10:31:01.9513179Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:01.9513509Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:01.9513941Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:01.9514270Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:01.9514610Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:31:01.9515150Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:01.9515482Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:01.9515896Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:01.9516240Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:01.9516575Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:31:01.9517042Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:01.9517356Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:01.9517783Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:01.9518127Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:01.9518450Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:31:01.9518943Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:01.9519464Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:01.9519981Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:01.9520349Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:31:01.9520727Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprq0n6e8t
2022-12-01T10:31:01.9521129Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprq0n6e8t/_remote_module_non_scriptable.py
2022-12-01T10:31:01.9521565Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:31:01.9521930Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:31:01.9522294Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_4b7n39t
2022-12-01T10:31:01.9522681Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptm0_k67z
2022-12-01T10:31:01.9523060Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_4b7n39t/_remote_module_non_scriptable.py
2022-12-01T10:31:01.9523460Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptm0_k67z/_remote_module_non_scriptable.py
2022-12-01T10:31:01.9523738Z ok (1.854s)
2022-12-01T10:31:01.9523842Z 
2022-12-01T10:31:01.9524043Z ----------------------------------------------------------------------
2022-12-01T10:31:01.9524272Z Ran 1 test in 1.854s
2022-12-01T10:31:01.9524384Z 
2022-12-01T10:31:01.9524445Z OK
2022-12-01T10:31:01.9524541Z 
2022-12-01T10:31:01.9524626Z Generating XML reports...
2022-12-01T10:31:01.9525073Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201101941.xml
2022-12-01T10:31:01.9525567Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:01.9526030Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:01.9526364Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:01.9526780Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:01.9527124Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:01.9527287Z 
2022-12-01T10:31:01.9527360Z Running tests...
2022-12-01T10:31:01.9527649Z ----------------------------------------------------------------------
2022-12-01T10:31:01.9527988Z   test_accumulate_gradients_no_sync_allreduce_hook (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9528476Z Runs multiple iterations on _test_accumulate_gradients_no_sync ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18819
2022-12-01T10:31:01.9528874Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18820
2022-12-01T10:31:01.9529189Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18821
2022-12-01T10:31:01.9529638Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:01.9529975Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:01.9530403Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:01.9530733Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:01.9531076Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:31:01.9531547Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:01.9531885Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:01.9532312Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:01.9532667Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:01.9533009Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:31:01.9533466Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:01.9533795Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:01.9534263Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:01.9534630Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:01.9534969Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:31:01.9535525Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:01.9536053Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:01.9536560Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:01.9536946Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:31:01.9537324Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplruyb79m
2022-12-01T10:31:01.9537731Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplruyb79m/_remote_module_non_scriptable.py
2022-12-01T10:31:01.9538101Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:31:01.9538449Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:31:01.9538818Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpry41kohf
2022-12-01T10:31:01.9539206Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbno2lgtg
2022-12-01T10:31:01.9539590Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpry41kohf/_remote_module_non_scriptable.py
2022-12-01T10:31:01.9539997Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbno2lgtg/_remote_module_non_scriptable.py
2022-12-01T10:31:01.9540487Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:31:01.9540856Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:31:01.9541214Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:31:01.9541473Z ok (1.855s)
2022-12-01T10:31:01.9541579Z 
2022-12-01T10:31:01.9541782Z ----------------------------------------------------------------------
2022-12-01T10:31:01.9542011Z Ran 1 test in 1.855s
2022-12-01T10:31:01.9542124Z 
2022-12-01T10:31:01.9542187Z OK
2022-12-01T10:31:01.9542281Z 
2022-12-01T10:31:01.9542366Z Generating XML reports...
2022-12-01T10:31:01.9542814Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201101944.xml
2022-12-01T10:31:01.9543312Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:01.9543779Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:01.9544113Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:01.9544526Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:01.9544879Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:01.9545044Z 
2022-12-01T10:31:01.9545117Z Running tests...
2022-12-01T10:31:01.9545405Z ----------------------------------------------------------------------
2022-12-01T10:31:01.9545753Z   test_accumulate_gradients_no_sync_allreduce_with_then_hook (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9546193Z Runs multiple iterations on _test_accumulate_gradients_no_sync using allreduce ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18872
2022-12-01T10:31:01.9546606Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18873
2022-12-01T10:31:01.9546923Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18874
2022-12-01T10:31:01.9547439Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:01.9547815Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:01.9548234Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:01.9548585Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:01.9548923Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:31:01.9549394Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:01.9549710Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:01.9550139Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:01.9550485Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:01.9550822Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:31:01.9551274Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:01.9551605Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:01.9552032Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:01.9552359Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:01.9552697Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:31:01.9553192Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:01.9553719Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:01.9554226Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:01.9554690Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:31:01.9555042Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:31:01.9555384Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:31:01.9555740Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_9k_jkzi
2022-12-01T10:31:01.9556134Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg6cu220j
2022-12-01T10:31:01.9556519Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuql_a613
2022-12-01T10:31:01.9556903Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_9k_jkzi/_remote_module_non_scriptable.py
2022-12-01T10:31:01.9557311Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg6cu220j/_remote_module_non_scriptable.py
2022-12-01T10:31:01.9557713Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuql_a613/_remote_module_non_scriptable.py
2022-12-01T10:31:01.9558091Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:31:01.9558435Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:31:01.9558790Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:31:01.9559048Z ok (1.753s)
2022-12-01T10:31:01.9559151Z 
2022-12-01T10:31:01.9559340Z ----------------------------------------------------------------------
2022-12-01T10:31:01.9559585Z Ran 1 test in 1.754s
2022-12-01T10:31:01.9559700Z 
2022-12-01T10:31:01.9559761Z OK
2022-12-01T10:31:01.9559853Z 
2022-12-01T10:31:01.9559975Z Generating XML reports...
2022-12-01T10:31:01.9560427Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201101947.xml
2022-12-01T10:31:01.9560960Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:01.9561423Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:01.9561741Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:01.9562168Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:01.9562513Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:01.9562675Z 
2022-12-01T10:31:01.9562749Z Running tests...
2022-12-01T10:31:01.9563039Z ----------------------------------------------------------------------
2022-12-01T10:31:01.9563373Z   test_accumulate_gradients_no_sync_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-12-01T10:31:01.9563787Z Runs _test_accumulate_gradients_no_sync using default inputs ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18925
2022-12-01T10:31:01.9564179Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18926
2022-12-01T10:31:01.9564497Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18927
2022-12-01T10:31:01.9564949Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:01.9565282Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:01.9565700Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:01.9566048Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:01.9566392Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:31:01.9566860Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:01.9567176Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:01.9567604Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:01.9567950Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:01.9568274Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:31:01.9568741Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:01.9569067Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:01.9569497Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:01.9569829Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:01.9570168Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:31:01.9570663Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:01.9571181Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:01.9571685Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:01.9572071Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:31:01.9572418Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:31:01.9572785Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:31:01.9573164Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwdk0uvxx
2022-12-01T10:31:01.9573602Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwdk0uvxx/_remote_module_non_scriptable.py
2022-12-01T10:31:01.9574000Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfiupj97p
2022-12-01T10:31:01.9574375Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwwv07f7k
2022-12-01T10:31:01.9574776Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfiupj97p/_remote_module_non_scriptable.py
2022-12-01T10:31:01.9575184Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwwv07f7k/_remote_module_non_scriptable.py
2022-12-01T10:31:01.9575558Z ok (1.854s)
2022-12-01T10:31:01.9575650Z 
2022-12-01T10:31:01.9575857Z ----------------------------------------------------------------------
2022-12-01T10:31:01.9575936Z Ran 1 test in 1.854s
2022-12-01T10:31:01.9575943Z 
2022-12-01T10:31:01.9576006Z OK
2022-12-01T10:31:01.9576012Z 
2022-12-01T10:31:01.9576098Z Generating XML reports...
2022-12-01T10:31:01.9576443Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201101950.xml
2022-12-01T10:31:01.9576677Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:01.9576952Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:01.9577064Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:01.9577347Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:01.9577484Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:01.9577488Z 
2022-12-01T10:31:01.9577562Z Running tests...
2022-12-01T10:31:01.9577761Z ----------------------------------------------------------------------
2022-12-01T10:31:01.9577988Z   test_all_gather (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18978
2022-12-01T10:31:01.9578146Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18979
2022-12-01T10:31:01.9578296Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 18980
2022-12-01T10:31:01.9578572Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:01.9578682Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:01.9578964Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:01.9579099Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:01.9579281Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:31:01.9579555Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:01.9579682Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:01.9579959Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:01.9580094Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:01.9580258Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:31:01.9580704Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:01.9580979Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:01.9581183Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:01.9581471Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:01.9581642Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:01.9581821Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:31:01.9582120Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:01.9582416Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:01.9582567Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:31:01.9582730Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:31:01.9582891Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:31:01.9583142Z STAGE:2022-12-01 10:19:55 18978:18978 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:01.9583390Z STAGE:2022-12-01 10:19:55 18979:18979 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:01.9583635Z STAGE:2022-12-01 10:19:55 18980:18980 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:01.9584207Z STAGE:2022-12-01 10:19:55 18978:18978 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 10:19:55 18980:18980 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 10:19:55 18979:18979 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:01.9584213Z 
2022-12-01T10:31:01.9584217Z 
2022-12-01T10:31:01.9584767Z STAGE:2022-12-01 10:19:55 18980:18980 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-12-01 10:19:55 18978:18978 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-12-01 10:19:55 18979:18979 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:01.9584775Z 
2022-12-01T10:31:01.9584779Z 
2022-12-01T10:31:01.9585187Z STAGE:2022-12-01 10:19:55 18978:18978 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 10:19:55 18979:18979 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:01.9585192Z 
2022-12-01T10:31:01.9585440Z STAGE:2022-12-01 10:19:55 18980:18980 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:01.9585682Z STAGE:2022-12-01 10:19:55 18979:18979 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:01.9585926Z STAGE:2022-12-01 10:19:55 18978:18978 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:01.9586168Z STAGE:2022-12-01 10:19:55 18980:18980 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:01.9586564Z STAGE:2022-12-01 10:19:55 18978:18978 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 10:19:55 18979:18979 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:01.9586584Z 
2022-12-01T10:31:01.9586815Z STAGE:2022-12-01 10:19:55 18980:18980 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:01.9586887Z ok (1.852s)
2022-12-01T10:31:01.9586892Z 
2022-12-01T10:31:01.9587089Z ----------------------------------------------------------------------
2022-12-01T10:31:01.9587166Z Ran 1 test in 1.853s
2022-12-01T10:31:01.9587170Z 
2022-12-01T10:31:01.9587234Z OK
2022-12-01T10:31:01.9587239Z 
2022-12-01T10:31:01.9587323Z Generating XML reports...
2022-12-01T10:31:01.9587664Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201101954.xml
2022-12-01T10:31:01.9587897Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:01.9588192Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:01.9588347Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:01.9588629Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:01.9588766Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:01.9588770Z 
2022-12-01T10:31:01.9588844Z Running tests...
2022-12-01T10:31:01.9589039Z ----------------------------------------------------------------------
2022-12-01T10:31:01.9589286Z   test_all_gather_coalesced_complex (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19034
2022-12-01T10:31:01.9589444Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19035
2022-12-01T10:31:01.9589593Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19036
2022-12-01T10:31:01.9589854Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:01.9589980Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:01.9590260Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:01.9590398Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:01.9590575Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:31:01.9590846Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:01.9590968Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:01.9591246Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:01.9591369Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:01.9591546Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:31:01.9591813Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:01.9591936Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:01.9592211Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:01.9592345Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:01.9592517Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:31:01.9592818Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:01.9593118Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:01.9593401Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:01.9593569Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:31:01.9593816Z STAGE:2022-12-01 10:19:59 19035:19035 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:01.9593981Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:31:01.9594144Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:31:01.9594390Z STAGE:2022-12-01 10:19:59 19034:19034 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:01.9594635Z STAGE:2022-12-01 10:19:59 19036:19036 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:01.9595076Z STAGE:2022-12-01 10:19:59 19036:19036 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 10:19:59 19034:19034 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:01.9595109Z 
2022-12-01T10:31:01.9595357Z STAGE:2022-12-01 10:19:59 19035:19035 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:01.9595758Z STAGE:2022-12-01 10:19:59 19036:19036 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-12-01 10:19:59 19034:19034 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:01.9595765Z 
2022-12-01T10:31:01.9596005Z STAGE:2022-12-01 10:19:59 19035:19035 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:01.9596235Z STAGE:2022-12-01 10:19:59 19036:19036 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:01.9596483Z STAGE:2022-12-01 10:19:59 19034:19034 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:01.9596731Z STAGE:2022-12-01 10:19:59 19035:19035 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:01.9596976Z STAGE:2022-12-01 10:19:59 19036:19036 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:01.9597217Z STAGE:2022-12-01 10:19:59 19034:19034 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:01.9597455Z STAGE:2022-12-01 10:19:59 19035:19035 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:01.9597698Z STAGE:2022-12-01 10:19:59 19035:19035 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:01.9597944Z STAGE:2022-12-01 10:19:59 19034:19034 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:01.9598189Z STAGE:2022-12-01 10:19:59 19036:19036 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:01.9598753Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:2442: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-12-01T10:31:01.9598910Z   "torch.distributed.all_gather_coalesced will be deprecated. If you must "
2022-12-01T10:31:01.9599470Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:2442: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-12-01T10:31:01.9599620Z   "torch.distributed.all_gather_coalesced will be deprecated. If you must "
2022-12-01T10:31:01.9600175Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:2442: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-12-01T10:31:01.9600328Z   "torch.distributed.all_gather_coalesced will be deprecated. If you must "
2022-12-01T10:31:01.9600399Z ok (1.952s)
2022-12-01T10:31:01.9600405Z 
2022-12-01T10:31:01.9600603Z ----------------------------------------------------------------------
2022-12-01T10:31:01.9600680Z Ran 1 test in 1.952s
2022-12-01T10:31:01.9600685Z 
2022-12-01T10:31:01.9600746Z OK
2022-12-01T10:31:01.9600750Z 
2022-12-01T10:31:01.9600821Z Generating XML reports...
2022-12-01T10:31:01.9601163Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201101957.xml
2022-12-01T10:31:01.9601397Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:01.9601671Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:01.9601796Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:01.9602111Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:01.9602278Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:01.9602283Z 
2022-12-01T10:31:01.9602356Z Running tests...
2022-12-01T10:31:01.9602553Z ----------------------------------------------------------------------
2022-12-01T10:31:01.9602786Z   test_all_gather_coalesced_full_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19090
2022-12-01T10:31:01.9602943Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19091
2022-12-01T10:31:01.9603094Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19092
2022-12-01T10:31:01.9603367Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:01.9603491Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:01.9603776Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:01.9603916Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:01.9604099Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:31:01.9604361Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:01.9604484Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:01.9604767Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:01.9604903Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:01.9605083Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:31:01.9605356Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:01.9605481Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:01.9605759Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:01.9605894Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:01.9606058Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:31:01.9606446Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:01.9606759Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:01.9607060Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:01.9607297Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:31:01.9607466Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:31:01.9607627Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:31:01.9607802Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T10:31:01.9607972Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T10:31:01.9608123Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-12-01T10:31:01.9608419Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:31:01.9608753Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:31:01.9609048Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:31:01.9609329Z STAGE:2022-12-01 10:20:02 19090:19090 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:01.9609576Z STAGE:2022-12-01 10:20:02 19091:19091 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:01.9609820Z STAGE:2022-12-01 10:20:02 19092:19092 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:01.9610235Z STAGE:2022-12-01 10:20:02 19091:19091 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 10:20:02 19090:19090 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:01.9610240Z 
2022-12-01T10:31:01.9610488Z STAGE:2022-12-01 10:20:02 19092:19092 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:01.9610892Z STAGE:2022-12-01 10:20:02 19092:19092 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-12-01 10:20:02 19091:19091 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:01.9610899Z 
2022-12-01T10:31:01.9611141Z STAGE:2022-12-01 10:20:02 19090:19090 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:01.9611375Z STAGE:2022-12-01 10:20:02 19092:19092 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:01.9611784Z STAGE:2022-12-01 10:20:02 19091:19091 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 10:20:02 19090:19090 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:01.9611790Z 
2022-12-01T10:31:01.9612031Z STAGE:2022-12-01 10:20:02 19091:19091 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:01.9612271Z STAGE:2022-12-01 10:20:02 19092:19092 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:01.9612512Z STAGE:2022-12-01 10:20:02 19090:19090 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:01.9612761Z STAGE:2022-12-01 10:20:02 19091:19091 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:01.9613008Z STAGE:2022-12-01 10:20:02 19092:19092 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:01.9613255Z STAGE:2022-12-01 10:20:02 19090:19090 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:01.9613822Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:2442: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-12-01T10:31:01.9613976Z   "torch.distributed.all_gather_coalesced will be deprecated. If you must "
2022-12-01T10:31:01.9614538Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:2442: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-12-01T10:31:01.9614676Z   "torch.distributed.all_gather_coalesced will be deprecated. If you must "
2022-12-01T10:31:01.9615304Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:2442: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-12-01T10:31:01.9615458Z   "torch.distributed.all_gather_coalesced will be deprecated. If you must "
2022-12-01T10:31:01.9615527Z ok (1.973s)
2022-12-01T10:31:01.9615532Z 
2022-12-01T10:31:01.9615734Z ----------------------------------------------------------------------
2022-12-01T10:31:01.9615810Z Ran 1 test in 1.973s
2022-12-01T10:31:01.9615815Z 
2022-12-01T10:31:01.9615876Z OK
2022-12-01T10:31:01.9615880Z 
2022-12-01T10:31:01.9616001Z Generating XML reports...
2022-12-01T10:31:01.9616374Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102000.xml
2022-12-01T10:31:01.9616597Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:01.9616871Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:01.9616996Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:01.9617276Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:01.9617413Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:01.9617418Z 
2022-12-01T10:31:01.9617492Z Running tests...
2022-12-01T10:31:01.9617687Z ----------------------------------------------------------------------
2022-12-01T10:31:01.9617933Z   test_all_gather_coalesced_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19155
2022-12-01T10:31:01.9618095Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19156
2022-12-01T10:31:01.9618234Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19157
2022-12-01T10:31:01.9618510Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:01.9618636Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:01.9618919Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:01.9619055Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:01.9619235Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:31:01.9619507Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:01.9619635Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:01.9619900Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:01.9620034Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:01.9620212Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:31:01.9620617Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:01.9620741Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:01.9621021Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:01.9621158Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:01.9621335Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:31:01.9621635Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:01.9621919Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:01.9622219Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:01.9622385Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:31:01.9622547Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:31:01.9622707Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:31:01.9622943Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T10:31:01.9623163Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-12-01T10:31:01.9623330Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T10:31:01.9623626Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:31:01.9623909Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:31:01.9624159Z STAGE:2022-12-01 10:20:05 19156:19156 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:01.9624452Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:31:01.9624697Z STAGE:2022-12-01 10:20:05 19157:19157 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:01.9624949Z STAGE:2022-12-01 10:20:05 19157:19157 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:01.9625198Z STAGE:2022-12-01 10:20:05 19156:19156 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:01.9625443Z STAGE:2022-12-01 10:20:05 19157:19157 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:01.9625683Z STAGE:2022-12-01 10:20:05 19156:19156 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:01.9625929Z STAGE:2022-12-01 10:20:05 19157:19157 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:01.9626160Z STAGE:2022-12-01 10:20:05 19156:19156 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:01.9626401Z STAGE:2022-12-01 10:20:05 19157:19157 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:01.9626639Z STAGE:2022-12-01 10:20:05 19156:19156 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:01.9626885Z STAGE:2022-12-01 10:20:05 19156:19156 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:01.9627129Z STAGE:2022-12-01 10:20:05 19157:19157 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:01.9627693Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:2442: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-12-01T10:31:01.9627847Z   "torch.distributed.all_gather_coalesced will be deprecated. If you must "
2022-12-01T10:31:01.9628416Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:2442: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-12-01T10:31:01.9628566Z   "torch.distributed.all_gather_coalesced will be deprecated. If you must "
2022-12-01T10:31:01.9628637Z ok (1.951s)
2022-12-01T10:31:01.9628641Z 
2022-12-01T10:31:01.9628826Z ----------------------------------------------------------------------
2022-12-01T10:31:01.9628903Z Ran 1 test in 1.951s
2022-12-01T10:31:01.9628908Z 
2022-12-01T10:31:01.9628969Z OK
2022-12-01T10:31:01.9628973Z 
2022-12-01T10:31:01.9629059Z Generating XML reports...
2022-12-01T10:31:01.9629400Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102004.xml
2022-12-01T10:31:01.9629637Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:01.9629912Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:01.9630037Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:01.9630364Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:01.9630515Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:01.9630521Z 
2022-12-01T10:31:01.9630596Z Running tests...
2022-12-01T10:31:01.9630793Z ----------------------------------------------------------------------
2022-12-01T10:31:01.9631038Z   test_all_gather_coalesced_simple (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19216
2022-12-01T10:31:01.9631197Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19217
2022-12-01T10:31:01.9631349Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19218
2022-12-01T10:31:01.9631627Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:01.9631753Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:01.9632025Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:01.9632160Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:01.9632338Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:31:01.9632606Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:01.9632730Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:01.9633010Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:01.9633145Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:01.9633324Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:31:01.9633597Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:01.9633710Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:01.9633990Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:01.9634126Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:01.9634303Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:31:01.9634602Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:01.9634904Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:01.9635204Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:01.9635371Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:31:01.9635534Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:31:01.9635680Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:31:01.9635928Z STAGE:2022-12-01 10:20:09 19217:19217 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:01.9636176Z STAGE:2022-12-01 10:20:09 19218:19218 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:01.9636418Z STAGE:2022-12-01 10:20:09 19216:19216 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:01.9636666Z STAGE:2022-12-01 10:20:09 19216:19216 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:01.9637109Z STAGE:2022-12-01 10:20:09 19218:19218 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 10:20:09 19217:19217 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:01.9637141Z 
2022-12-01T10:31:01.9637384Z STAGE:2022-12-01 10:20:09 19216:19216 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:01.9637786Z STAGE:2022-12-01 10:20:09 19218:19218 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-12-01 10:20:09 19217:19217 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:01.9637792Z 
2022-12-01T10:31:01.9638037Z STAGE:2022-12-01 10:20:09 19216:19216 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:01.9638442Z STAGE:2022-12-01 10:20:09 19218:19218 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 10:20:09 19217:19217 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:01.9638448Z 
2022-12-01T10:31:01.9638689Z STAGE:2022-12-01 10:20:09 19216:19216 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:01.9638921Z STAGE:2022-12-01 10:20:09 19217:19217 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:01.9639160Z STAGE:2022-12-01 10:20:09 19218:19218 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:01.9639405Z STAGE:2022-12-01 10:20:09 19216:19216 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:01.9639809Z STAGE:2022-12-01 10:20:09 19218:19218 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 10:20:09 19217:19217 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:01.9639815Z 
2022-12-01T10:31:01.9640379Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:2442: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-12-01T10:31:01.9640532Z   "torch.distributed.all_gather_coalesced will be deprecated. If you must "
2022-12-01T10:31:01.9641091Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:2442: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-12-01T10:31:01.9641243Z   "torch.distributed.all_gather_coalesced will be deprecated. If you must "
2022-12-01T10:31:01.9641802Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:2442: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-12-01T10:31:01.9641951Z   "torch.distributed.all_gather_coalesced will be deprecated. If you must "
2022-12-01T10:31:01.9642020Z ok (1.858s)
2022-12-01T10:31:01.9642025Z 
2022-12-01T10:31:01.9642226Z ----------------------------------------------------------------------
2022-12-01T10:31:01.9642291Z Ran 1 test in 1.858s
2022-12-01T10:31:01.9642297Z 
2022-12-01T10:31:01.9642358Z OK
2022-12-01T10:31:01.9642363Z 
2022-12-01T10:31:01.9642448Z Generating XML reports...
2022-12-01T10:31:01.9642791Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102007.xml
2022-12-01T10:31:01.9643026Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:01.9643304Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:01.9643432Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:01.9643714Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:01.9643869Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:01.9643888Z 
2022-12-01T10:31:01.9643976Z Running tests...
2022-12-01T10:31:01.9644173Z ----------------------------------------------------------------------
2022-12-01T10:31:01.9644420Z   test_all_gather_coalesced_with_empty (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19272
2022-12-01T10:31:01.9644576Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19273
2022-12-01T10:31:01.9644726Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19274
2022-12-01T10:31:01.9645001Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:01.9645127Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:01.9645407Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:01.9645533Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:01.9645713Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:31:01.9645983Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:01.9646106Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:01.9646383Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:01.9646519Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:01.9646690Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:31:01.9646964Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:01.9647089Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:01.9647364Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:01.9647501Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:01.9647675Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:31:01.9647959Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:01.9648257Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:01.9648553Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:01.9648720Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:31:01.9648888Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:31:01.9649140Z STAGE:2022-12-01 10:20:12 19274:19274 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:01.9649387Z STAGE:2022-12-01 10:20:12 19273:19273 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:01.9649550Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:31:01.9649791Z STAGE:2022-12-01 10:20:12 19272:19272 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:01.9650024Z STAGE:2022-12-01 10:20:12 19272:19272 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:01.9650272Z STAGE:2022-12-01 10:20:12 19273:19273 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:01.9650519Z STAGE:2022-12-01 10:20:12 19274:19274 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:01.9651111Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:2442: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-12-01T10:31:01.9651293Z   "torch.distributed.all_gather_coalesced will be deprecated. If you must "
2022-12-01T10:31:01.9651854Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:2442: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-12-01T10:31:01.9652005Z   "torch.distributed.all_gather_coalesced will be deprecated. If you must "
2022-12-01T10:31:01.9652560Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:2442: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-12-01T10:31:01.9652712Z   "torch.distributed.all_gather_coalesced will be deprecated. If you must "
2022-12-01T10:31:01.9652781Z ok (1.956s)
2022-12-01T10:31:01.9652786Z 
2022-12-01T10:31:01.9652982Z ----------------------------------------------------------------------
2022-12-01T10:31:01.9653044Z Ran 1 test in 1.956s
2022-12-01T10:31:01.9653049Z 
2022-12-01T10:31:01.9653111Z OK
2022-12-01T10:31:01.9653115Z 
2022-12-01T10:31:01.9653202Z Generating XML reports...
2022-12-01T10:31:01.9653544Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102011.xml
2022-12-01T10:31:01.9653778Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:01.9654051Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:01.9654179Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:01.9654461Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:01.9654583Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:01.9654600Z 
2022-12-01T10:31:01.9654661Z Running tests...
2022-12-01T10:31:01.9654854Z ----------------------------------------------------------------------
2022-12-01T10:31:01.9655086Z   test_all_gather_complex (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19328
2022-12-01T10:31:01.9655354Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19329
2022-12-01T10:31:01.9655511Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19330
2022-12-01T10:31:01.9655789Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:01.9655914Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:01.9656198Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:01.9656321Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:01.9656501Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:31:01.9656771Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:01.9656894Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:01.9657173Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:01.9657308Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:01.9657522Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:31:01.9657821Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:01.9657930Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:01.9658207Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:01.9658339Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:01.9658514Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:31:01.9658810Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:01.9659108Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:01.9659406Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:01.9659572Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:31:01.9659736Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:31:01.9659883Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:31:01.9660134Z STAGE:2022-12-01 10:20:16 19328:19328 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:01.9660525Z STAGE:2022-12-01 10:20:16 19330:19330 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:01.9660776Z STAGE:2022-12-01 10:20:16 19329:19329 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:01.9661026Z STAGE:2022-12-01 10:20:16 19330:19330 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:01.9696780Z STAGE:2022-12-01 10:20:16 19329:19329 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 10:20:16 19328:19328 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:01.9696802Z 
2022-12-01T10:31:01.9697196Z STAGE:2022-12-01 10:20:16 19330:19330 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:01.9697603Z STAGE:2022-12-01 10:20:16 19328:19328 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-12-01 10:20:16 19329:19329 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:01.9697617Z 
2022-12-01T10:31:01.9697857Z STAGE:2022-12-01 10:20:16 19330:19330 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:01.9698266Z STAGE:2022-12-01 10:20:16 19328:19328 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 10:20:16 19329:19329 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:01.9698272Z 
2022-12-01T10:31:01.9698514Z STAGE:2022-12-01 10:20:16 19330:19330 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:01.9698910Z STAGE:2022-12-01 10:20:16 19328:19328 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-12-01 10:20:16 19329:19329 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:01.9698916Z 
2022-12-01T10:31:01.9699165Z STAGE:2022-12-01 10:20:16 19328:19328 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:01.9699401Z STAGE:2022-12-01 10:20:16 19330:19330 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:01.9699646Z STAGE:2022-12-01 10:20:16 19329:19329 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:01.9699709Z ok (1.853s)
2022-12-01T10:31:01.9699715Z 
2022-12-01T10:31:01.9699915Z ----------------------------------------------------------------------
2022-12-01T10:31:01.9699985Z Ran 1 test in 1.853s
2022-12-01T10:31:01.9699990Z 
2022-12-01T10:31:01.9700173Z OK
2022-12-01T10:31:01.9700179Z 
2022-12-01T10:31:01.9700266Z Generating XML reports...
2022-12-01T10:31:01.9700842Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102014.xml
2022-12-01T10:31:01.9701070Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:01.9701344Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:01.9701472Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:01.9701755Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:01.9701888Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:01.9701893Z 
2022-12-01T10:31:01.9701966Z Running tests...
2022-12-01T10:31:01.9702152Z ----------------------------------------------------------------------
2022-12-01T10:31:01.9702339Z   test_all_gather_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all gather (0.002s)
2022-12-01T10:31:01.9702346Z 
2022-12-01T10:31:01.9702539Z ----------------------------------------------------------------------
2022-12-01T10:31:01.9702610Z Ran 1 test in 0.002s
2022-12-01T10:31:01.9702615Z 
2022-12-01T10:31:01.9702687Z OK (skipped=1)
2022-12-01T10:31:01.9702691Z 
2022-12-01T10:31:01.9702774Z Generating XML reports...
2022-12-01T10:31:01.9703108Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102017.xml
2022-12-01T10:31:01.9703338Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:01.9703601Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:01.9703719Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:01.9704001Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:01.9704132Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:01.9704137Z 
2022-12-01T10:31:01.9704209Z Running tests...
2022-12-01T10:31:01.9704403Z ----------------------------------------------------------------------
2022-12-01T10:31:01.9704597Z   test_all_gather_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all gather (0.002s)
2022-12-01T10:31:01.9704602Z 
2022-12-01T10:31:01.9704786Z ----------------------------------------------------------------------
2022-12-01T10:31:01.9704862Z Ran 1 test in 0.002s
2022-12-01T10:31:01.9704866Z 
2022-12-01T10:31:01.9704924Z OK (skipped=1)
2022-12-01T10:31:01.9704940Z 
2022-12-01T10:31:01.9705010Z Generating XML reports...
2022-12-01T10:31:01.9705344Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102019.xml
2022-12-01T10:31:01.9705576Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:01.9705846Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:01.9705968Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:01.9706242Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:01.9706376Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:01.9706381Z 
2022-12-01T10:31:01.9706451Z Running tests...
2022-12-01T10:31:01.9706632Z ----------------------------------------------------------------------
2022-12-01T10:31:01.9706860Z   test_all_gather_full_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19404
2022-12-01T10:31:01.9707060Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19405
2022-12-01T10:31:01.9707242Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19406
2022-12-01T10:31:01.9707514Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:01.9707629Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:01.9707907Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:01.9708033Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:01.9708200Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:31:01.9708475Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:01.9708592Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:01.9708871Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:01.9709007Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:01.9709181Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:31:01.9709449Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:01.9709565Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:01.9709844Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:01.9709965Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:01.9710135Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:31:01.9710437Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:01.9710727Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:01.9711018Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:01.9711176Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:31:01.9711340Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:31:01.9711499Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:31:01.9711666Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T10:31:01.9711829Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T10:31:01.9712119Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:31:01.9712293Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-12-01T10:31:01.9712588Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:31:01.9712878Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:31:01.9713121Z STAGE:2022-12-01 10:20:22 19404:19404 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:01.9713356Z STAGE:2022-12-01 10:20:22 19405:19405 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:01.9713588Z STAGE:2022-12-01 10:20:22 19406:19406 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:01.9714200Z STAGE:2022-12-01 10:20:22 19404:19404 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 10:20:22 19405:19405 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 10:20:22 19406:19406 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:01.9714233Z 
2022-12-01T10:31:01.9714237Z 
2022-12-01T10:31:01.9714636Z STAGE:2022-12-01 10:20:22 19404:19404 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-12-01 10:20:22 19405:19405 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:01.9714642Z 
2022-12-01T10:31:01.9714872Z STAGE:2022-12-01 10:20:22 19406:19406 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:01.9715118Z STAGE:2022-12-01 10:20:22 19404:19404 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:01.9715348Z STAGE:2022-12-01 10:20:22 19405:19405 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:01.9715582Z STAGE:2022-12-01 10:20:22 19406:19406 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:01.9715822Z STAGE:2022-12-01 10:20:22 19404:19404 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:01.9716061Z STAGE:2022-12-01 10:20:22 19405:19405 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:01.9716299Z STAGE:2022-12-01 10:20:22 19406:19406 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:01.9716544Z STAGE:2022-12-01 10:20:22 19404:19404 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:01.9716779Z STAGE:2022-12-01 10:20:22 19405:19405 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:01.9717016Z STAGE:2022-12-01 10:20:22 19406:19406 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:01.9717072Z ok (1.961s)
2022-12-01T10:31:01.9717082Z 
2022-12-01T10:31:01.9717269Z ----------------------------------------------------------------------
2022-12-01T10:31:01.9717345Z Ran 1 test in 1.961s
2022-12-01T10:31:01.9717350Z 
2022-12-01T10:31:01.9717405Z OK
2022-12-01T10:31:01.9717409Z 
2022-12-01T10:31:01.9717492Z Generating XML reports...
2022-12-01T10:31:01.9717828Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102020.xml
2022-12-01T10:31:01.9718059Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:01.9718328Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:01.9718451Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:01.9718720Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:01.9718850Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:01.9718857Z 
2022-12-01T10:31:01.9718929Z Running tests...
2022-12-01T10:31:01.9719126Z ----------------------------------------------------------------------
2022-12-01T10:31:01.9719348Z   test_all_gather_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19469
2022-12-01T10:31:01.9719499Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19470
2022-12-01T10:31:01.9719653Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19471
2022-12-01T10:31:01.9719921Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:01.9720032Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:01.9720310Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:01.9720470Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:01.9720645Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:31:01.9720946Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:01.9721067Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:01.9721343Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:01.9721474Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:01.9721645Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:31:01.9721904Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:01.9722027Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:01.9722299Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:01.9722433Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:01.9722602Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:31:01.9722895Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:01.9723191Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:01.9723482Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:01.9723638Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:31:01.9723792Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:31:01.9723965Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T10:31:01.9724134Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-12-01T10:31:01.9724292Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:31:01.9724458Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T10:31:01.9724749Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:31:01.9725035Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:31:01.9725321Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:31:01.9725569Z STAGE:2022-12-01 10:20:25 19470:19470 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:01.9725799Z STAGE:2022-12-01 10:20:25 19471:19471 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:01.9726042Z STAGE:2022-12-01 10:20:25 19471:19471 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:01.9726284Z STAGE:2022-12-01 10:20:25 19470:19470 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:01.9726513Z STAGE:2022-12-01 10:20:25 19471:19471 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:01.9726749Z STAGE:2022-12-01 10:20:25 19470:19470 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:01.9727155Z STAGE:2022-12-01 10:20:25 19471:19471 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 10:20:25 19470:19470 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:01.9727161Z 
2022-12-01T10:31:01.9727266Z ok (2.054s)
2022-12-01T10:31:01.9727271Z 
2022-12-01T10:31:01.9727488Z ----------------------------------------------------------------------
2022-12-01T10:31:01.9727558Z Ran 1 test in 2.055s
2022-12-01T10:31:01.9727564Z 
2022-12-01T10:31:01.9727611Z OK
2022-12-01T10:31:01.9727627Z 
2022-12-01T10:31:01.9727698Z Generating XML reports...
2022-12-01T10:31:01.9728035Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102023.xml
2022-12-01T10:31:01.9728264Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:01.9728535Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:01.9728658Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:01.9728932Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:01.9729071Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:01.9729079Z 
2022-12-01T10:31:01.9729151Z Running tests...
2022-12-01T10:31:01.9729333Z ----------------------------------------------------------------------
2022-12-01T10:31:01.9729543Z   test_all_gather_into_cat_tensor_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_gather_into_tensor (0.002s)
2022-12-01T10:31:01.9729548Z 
2022-12-01T10:31:01.9729736Z ----------------------------------------------------------------------
2022-12-01T10:31:01.9729805Z Ran 1 test in 0.002s
2022-12-01T10:31:01.9729810Z 
2022-12-01T10:31:01.9729879Z OK (skipped=1)
2022-12-01T10:31:01.9729883Z 
2022-12-01T10:31:01.9729960Z Generating XML reports...
2022-12-01T10:31:01.9730297Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102027.xml
2022-12-01T10:31:01.9730526Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:01.9730797Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:01.9730910Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:01.9731189Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:01.9731321Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:01.9731325Z 
2022-12-01T10:31:01.9731396Z Running tests...
2022-12-01T10:31:01.9731584Z ----------------------------------------------------------------------
2022-12-01T10:31:01.9731800Z   test_all_gather_into_stack_tensor_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_gather_into_tensor (0.002s)
2022-12-01T10:31:01.9731804Z 
2022-12-01T10:31:01.9731992Z ----------------------------------------------------------------------
2022-12-01T10:31:01.9732065Z Ran 1 test in 0.002s
2022-12-01T10:31:01.9732070Z 
2022-12-01T10:31:01.9732141Z OK (skipped=1)
2022-12-01T10:31:01.9732145Z 
2022-12-01T10:31:01.9732216Z Generating XML reports...
2022-12-01T10:31:01.9732546Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102028.xml
2022-12-01T10:31:01.9732777Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:01.9733042Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:01.9733162Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:01.9733438Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:01.9733571Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:01.9733576Z 
2022-12-01T10:31:01.9733642Z Running tests...
2022-12-01T10:31:01.9733855Z ----------------------------------------------------------------------
2022-12-01T10:31:01.9734091Z   test_all_gather_multigpu (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl backend supports allgather multigpu (0.002s)
2022-12-01T10:31:01.9734095Z 
2022-12-01T10:31:01.9734281Z ----------------------------------------------------------------------
2022-12-01T10:31:01.9734354Z Ran 1 test in 0.002s
2022-12-01T10:31:01.9734360Z 
2022-12-01T10:31:01.9734425Z OK (skipped=1)
2022-12-01T10:31:01.9734429Z 
2022-12-01T10:31:01.9734507Z Generating XML reports...
2022-12-01T10:31:01.9734836Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102030.xml
2022-12-01T10:31:01.9735064Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:01.9735421Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:01.9735536Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:01.9735816Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:01.9735945Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:01.9735950Z 
2022-12-01T10:31:01.9736016Z Running tests...
2022-12-01T10:31:01.9736207Z ----------------------------------------------------------------------
2022-12-01T10:31:01.9736424Z   test_all_gather_multigpu_complex (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl backend supports allgather multigpu (0.002s)
2022-12-01T10:31:01.9736429Z 
2022-12-01T10:31:01.9736617Z ----------------------------------------------------------------------
2022-12-01T10:31:01.9736692Z Ran 1 test in 0.002s
2022-12-01T10:31:01.9736697Z 
2022-12-01T10:31:01.9736764Z OK (skipped=1)
2022-12-01T10:31:01.9736768Z 
2022-12-01T10:31:01.9736840Z Generating XML reports...
2022-12-01T10:31:01.9737182Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102031.xml
2022-12-01T10:31:01.9737446Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:01.9737817Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:01.9737937Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:01.9738211Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:01.9738345Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:01.9738350Z 
2022-12-01T10:31:01.9738419Z Running tests...
2022-12-01T10:31:01.9738607Z ----------------------------------------------------------------------
2022-12-01T10:31:01.9738837Z   test_all_gather_object_default_pg (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19570
2022-12-01T10:31:01.9738997Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19571
2022-12-01T10:31:01.9739151Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19572
2022-12-01T10:31:01.9739417Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:01.9739534Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:01.9739817Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:01.9739946Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:01.9740122Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:31:01.9740566Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:01.9740722Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:01.9741004Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:01.9741135Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:01.9741316Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:31:01.9741589Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:01.9741715Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:01.9741989Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:01.9742119Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:01.9742284Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:31:01.9742585Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:01.9742879Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:01.9743169Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:01.9743330Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:31:01.9743490Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:31:01.9743650Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:31:01.9743713Z ok (1.857s)
2022-12-01T10:31:01.9743718Z 
2022-12-01T10:31:01.9743911Z ----------------------------------------------------------------------
2022-12-01T10:31:01.9743974Z Ran 1 test in 1.857s
2022-12-01T10:31:01.9743979Z 
2022-12-01T10:31:01.9744039Z OK
2022-12-01T10:31:01.9744044Z 
2022-12-01T10:31:01.9744124Z Generating XML reports...
2022-12-01T10:31:01.9744461Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102032.xml
2022-12-01T10:31:01.9744686Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:01.9744959Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:01.9745079Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:01.9745362Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:01.9745484Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:01.9745502Z 
2022-12-01T10:31:01.9745562Z Running tests...
2022-12-01T10:31:01.9745759Z ----------------------------------------------------------------------
2022-12-01T10:31:01.9745994Z   test_all_gather_object_subgroup (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19623
2022-12-01T10:31:01.9746145Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19624
2022-12-01T10:31:01.9746299Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19625
2022-12-01T10:31:01.9746567Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:01.9746689Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:01.9746964Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:01.9747118Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:01.9747318Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:31:01.9747617Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:01.9747884Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:01.9748004Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:01.9748276Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:01.9748404Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:01.9748578Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:31:01.9748845Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:01.9748956Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:01.9749234Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:01.9749362Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:01.9749536Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:31:01.9749832Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:01.9750129Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:01.9750293Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:31:01.9750458Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:31:01.9750618Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:31:01.9750779Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T10:31:01.9750953Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-12-01T10:31:01.9751119Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T10:31:01.9751413Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:31:01.9751704Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:31:01.9751994Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:31:01.9752164Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-12-01T10:31:01.9752339Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 2
2022-12-01T10:31:01.9752505Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-12-01T10:31:01.9752780Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:3 with 3 nodes.
2022-12-01T10:31:01.9753070Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 3 nodes.
2022-12-01T10:31:01.9753356Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 3 nodes.
2022-12-01T10:31:01.9753524Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 1
2022-12-01T10:31:01.9753724Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 0
2022-12-01T10:31:01.9753922Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 2
2022-12-01T10:31:01.9754210Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:4 with 3 nodes.
2022-12-01T10:31:01.9754493Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:4 with 3 nodes.
2022-12-01T10:31:01.9754786Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:4 with 3 nodes.
2022-12-01T10:31:01.9754842Z ok (2.055s)
2022-12-01T10:31:01.9754847Z 
2022-12-01T10:31:01.9755040Z ----------------------------------------------------------------------
2022-12-01T10:31:01.9755111Z Ran 1 test in 2.056s
2022-12-01T10:31:01.9755115Z 
2022-12-01T10:31:01.9755174Z OK
2022-12-01T10:31:01.9755179Z 
2022-12-01T10:31:01.9755259Z Generating XML reports...
2022-12-01T10:31:01.9755599Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102036.xml
2022-12-01T10:31:01.9755832Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:01.9756098Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:01.9756219Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:01.9756486Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:01.9756616Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:01.9756621Z 
2022-12-01T10:31:01.9756693Z Running tests...
2022-12-01T10:31:01.9756885Z ----------------------------------------------------------------------
2022-12-01T10:31:01.9757076Z   test_all_gather_v_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports all_gather_v (0.002s)
2022-12-01T10:31:01.9757084Z 
2022-12-01T10:31:01.9757273Z ----------------------------------------------------------------------
2022-12-01T10:31:01.9757346Z Ran 1 test in 0.002s
2022-12-01T10:31:01.9757351Z 
2022-12-01T10:31:01.9757418Z OK (skipped=1)
2022-12-01T10:31:01.9757423Z 
2022-12-01T10:31:01.9757493Z Generating XML reports...
2022-12-01T10:31:01.9757830Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102039.xml
2022-12-01T10:31:01.9758057Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:01.9758327Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:01.9758446Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:01.9758728Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:01.9758860Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:01.9758865Z 
2022-12-01T10:31:01.9758937Z Running tests...
2022-12-01T10:31:01.9759125Z ----------------------------------------------------------------------
2022-12-01T10:31:01.9759361Z   test_all_reduce_coalesced_full_group_max (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19722
2022-12-01T10:31:01.9759521Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19723
2022-12-01T10:31:01.9759676Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19724
2022-12-01T10:31:01.9759954Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:01.9760076Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:01.9760393Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:01.9760556Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:01.9760730Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:31:01.9760996Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:01.9761106Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:01.9761387Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:01.9761516Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:01.9761689Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:31:01.9761958Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:01.9762083Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:01.9762358Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:01.9762490Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:01.9762654Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:31:01.9762948Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:01.9763238Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:01.9763533Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:01.9763698Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:31:01.9763859Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:31:01.9764017Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:31:01.9764187Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T10:31:01.9764357Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-12-01T10:31:01.9764515Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T10:31:01.9764810Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:31:01.9765104Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:31:01.9765394Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:31:01.9765639Z STAGE:2022-12-01 10:20:42 19722:19722 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:01.9766040Z STAGE:2022-12-01 10:20:42 19724:19724 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-12-01 10:20:42 19723:19723 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:01.9766046Z 
2022-12-01T10:31:01.9766294Z STAGE:2022-12-01 10:20:42 19723:19723 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:01.9766701Z STAGE:2022-12-01 10:20:42 19724:19724 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 10:20:42 19722:19722 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:01.9766707Z 
2022-12-01T10:31:01.9766978Z STAGE:2022-12-01 10:20:42 19723:19723 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:01.9767369Z STAGE:2022-12-01 10:20:42 19724:19724 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-12-01 10:20:42 19722:19722 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:01.9767403Z 
2022-12-01T10:31:01.9767649Z STAGE:2022-12-01 10:20:42 19722:19722 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:01.9767892Z STAGE:2022-12-01 10:20:42 19723:19723 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:01.9768119Z STAGE:2022-12-01 10:20:42 19722:19722 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:01.9768355Z STAGE:2022-12-01 10:20:42 19723:19723 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:01.9768597Z STAGE:2022-12-01 10:20:42 19724:19724 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:01.9768838Z STAGE:2022-12-01 10:20:42 19724:19724 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:01.9769078Z STAGE:2022-12-01 10:20:42 19722:19722 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:01.9769321Z STAGE:2022-12-01 10:20:42 19724:19724 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:01.9769557Z STAGE:2022-12-01 10:20:42 19723:19723 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:01.9770132Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1578: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-12-01T10:31:01.9770282Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2022-12-01T10:31:01.9770832Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1578: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-12-01T10:31:01.9770972Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2022-12-01T10:31:01.9771529Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1578: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-12-01T10:31:01.9771672Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2022-12-01T10:31:01.9771738Z ok (1.957s)
2022-12-01T10:31:01.9771743Z 
2022-12-01T10:31:01.9771937Z ----------------------------------------------------------------------
2022-12-01T10:31:01.9772013Z Ran 1 test in 1.957s
2022-12-01T10:31:01.9772018Z 
2022-12-01T10:31:01.9772077Z OK
2022-12-01T10:31:01.9772081Z 
2022-12-01T10:31:01.9772162Z Generating XML reports...
2022-12-01T10:31:01.9772497Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102041.xml
2022-12-01T10:31:01.9772717Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:01.9772991Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:01.9773147Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:01.9773425Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:01.9773559Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:01.9773564Z 
2022-12-01T10:31:01.9773634Z Running tests...
2022-12-01T10:31:01.9773828Z ----------------------------------------------------------------------
2022-12-01T10:31:01.9774106Z   test_all_reduce_coalesced_full_group_min (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19787
2022-12-01T10:31:01.9774290Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19788
2022-12-01T10:31:01.9774431Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19789
2022-12-01T10:31:01.9774701Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:01.9774824Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:01.9775100Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:01.9775321Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:01.9775499Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:31:01.9775779Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:01.9775901Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:01.9776179Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:01.9776300Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:01.9776474Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:31:01.9776766Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:01.9777036Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:01.9777156Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:01.9777435Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:01.9777567Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:01.9777742Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:31:01.9778025Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:01.9778313Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:01.9778480Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:31:01.9778641Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:31:01.9778802Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:31:01.9778975Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T10:31:01.9779151Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-12-01T10:31:01.9779319Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T10:31:01.9779617Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:31:01.9779900Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:31:01.9780190Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:31:01.9780710Z STAGE:2022-12-01 10:20:46 19789:19789 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-12-01 10:20:46 19787:19787 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:01.9781014Z STAGE:2022-12-01 10:20:46 19788:19788 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:01.9781054Z 
2022-12-01T10:31:01.9781304Z STAGE:2022-12-01 10:20:46 19787:19787 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:01.9781713Z STAGE:2022-12-01 10:20:46 19788:19788 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 10:20:46 19789:19789 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:01.9781719Z 
2022-12-01T10:31:01.9782265Z STAGE:2022-12-01 10:20:46 19789:19789 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-12-01 10:20:46 19787:19787 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-12-01 10:20:46 19788:19788 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:01.9782271Z 
2022-12-01T10:31:01.9782275Z 
2022-12-01T10:31:01.9782524Z STAGE:2022-12-01 10:20:46 19788:19788 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:01.9782926Z STAGE:2022-12-01 10:20:46 19787:19787 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 10:20:46 19789:19789 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:01.9782934Z 
2022-12-01T10:31:01.9783176Z STAGE:2022-12-01 10:20:46 19787:19787 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:01.9783410Z STAGE:2022-12-01 10:20:46 19788:19788 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:01.9783653Z STAGE:2022-12-01 10:20:46 19789:19789 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:01.9783896Z STAGE:2022-12-01 10:20:46 19789:19789 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:01.9784134Z STAGE:2022-12-01 10:20:46 19788:19788 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:01.9784360Z STAGE:2022-12-01 10:20:46 19787:19787 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:01.9784924Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1578: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-12-01T10:31:01.9785079Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2022-12-01T10:31:01.9785629Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1578: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-12-01T10:31:01.9785779Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2022-12-01T10:31:01.9786332Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1578: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-12-01T10:31:01.9786478Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2022-12-01T10:31:01.9786546Z ok (1.952s)
2022-12-01T10:31:01.9786551Z 
2022-12-01T10:31:01.9786747Z ----------------------------------------------------------------------
2022-12-01T10:31:01.9786823Z Ran 1 test in 1.952s
2022-12-01T10:31:01.9786829Z 
2022-12-01T10:31:01.9786877Z OK
2022-12-01T10:31:01.9786892Z 
2022-12-01T10:31:01.9786964Z Generating XML reports...
2022-12-01T10:31:01.9787306Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102044.xml
2022-12-01T10:31:01.9787535Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:01.9787839Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:01.9787990Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:01.9788271Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:01.9788405Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:01.9788411Z 
2022-12-01T10:31:01.9788482Z Running tests...
2022-12-01T10:31:01.9788664Z ----------------------------------------------------------------------
2022-12-01T10:31:01.9788916Z   test_all_reduce_coalesced_full_group_product (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19852
2022-12-01T10:31:01.9789072Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19853
2022-12-01T10:31:01.9789226Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19854
2022-12-01T10:31:01.9789499Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:01.9789619Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:01.9789898Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:01.9790033Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:01.9790196Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:31:01.9790468Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:01.9790588Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:01.9790868Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:01.9791000Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:01.9791175Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:31:01.9791445Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:01.9791564Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:01.9791838Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:01.9791959Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:01.9792134Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:31:01.9792428Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:01.9792722Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:01.9793016Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:01.9793176Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:31:01.9793339Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:31:01.9793498Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:31:01.9793671Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T10:31:01.9793831Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-12-01T10:31:01.9793996Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T10:31:01.9794343Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:31:01.9794660Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:31:01.9794947Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:31:01.9795195Z STAGE:2022-12-01 10:20:49 19852:19852 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:01.9795596Z STAGE:2022-12-01 10:20:49 19853:19853 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-12-01 10:20:49 19854:19854 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:01.9795602Z 
2022-12-01T10:31:01.9795849Z STAGE:2022-12-01 10:20:49 19854:19854 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:01.9796259Z STAGE:2022-12-01 10:20:49 19853:19853 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 10:20:49 19852:19852 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:01.9796267Z 
2022-12-01T10:31:01.9796512Z STAGE:2022-12-01 10:20:49 19853:19853 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:01.9796750Z STAGE:2022-12-01 10:20:49 19854:19854 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:01.9796974Z STAGE:2022-12-01 10:20:49 19852:19852 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:01.9797541Z STAGE:2022-12-01 10:20:49 19852:19852 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 10:20:49 19854:19854 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 10:20:49 19853:19853 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:01.9797547Z 
2022-12-01T10:31:01.9797566Z 
2022-12-01T10:31:01.9797799Z STAGE:2022-12-01 10:20:49 19854:19854 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:01.9798035Z STAGE:2022-12-01 10:20:49 19853:19853 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:01.9798275Z STAGE:2022-12-01 10:20:49 19852:19852 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:01.9798521Z STAGE:2022-12-01 10:20:49 19853:19853 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:01.9798763Z STAGE:2022-12-01 10:20:49 19852:19852 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:01.9799009Z STAGE:2022-12-01 10:20:49 19854:19854 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:01.9799569Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1578: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-12-01T10:31:01.9799720Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2022-12-01T10:31:01.9800272Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1578: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-12-01T10:31:01.9800413Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2022-12-01T10:31:01.9800960Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1578: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-12-01T10:31:01.9801102Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2022-12-01T10:31:01.9801172Z ok (2.056s)
2022-12-01T10:31:01.9801177Z 
2022-12-01T10:31:01.9801414Z ----------------------------------------------------------------------
2022-12-01T10:31:01.9801519Z Ran 1 test in 2.056s
2022-12-01T10:31:01.9801524Z 
2022-12-01T10:31:01.9801582Z OK
2022-12-01T10:31:01.9801587Z 
2022-12-01T10:31:01.9801671Z Generating XML reports...
2022-12-01T10:31:01.9802010Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102047.xml
2022-12-01T10:31:01.9802239Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:01.9802498Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:01.9802620Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:01.9802901Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:01.9803033Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:01.9803040Z 
2022-12-01T10:31:01.9803111Z Running tests...
2022-12-01T10:31:01.9803303Z ----------------------------------------------------------------------
2022-12-01T10:31:01.9803552Z   test_all_reduce_coalesced_full_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19917
2022-12-01T10:31:01.9803706Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19918
2022-12-01T10:31:01.9803857Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19919
2022-12-01T10:31:01.9804117Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:01.9804241Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:01.9804515Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:01.9804652Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:01.9804832Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:31:01.9805103Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:01.9805226Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:01.9805505Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:01.9805627Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:01.9805802Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:31:01.9806073Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:01.9806194Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:01.9806472Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:01.9806603Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:01.9806776Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:31:01.9807072Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:01.9807364Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:01.9807649Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:01.9807809Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:31:01.9808014Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:31:01.9808207Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:31:01.9808382Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T10:31:01.9808554Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-12-01T10:31:01.9808724Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T10:31:01.9809017Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:31:01.9809310Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:31:01.9809590Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:31:01.9809838Z STAGE:2022-12-01 10:20:53 19918:19918 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:01.9810082Z STAGE:2022-12-01 10:20:53 19917:19917 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:01.9810314Z STAGE:2022-12-01 10:20:53 19919:19919 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:01.9810563Z STAGE:2022-12-01 10:20:53 19918:19918 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:01.9810809Z STAGE:2022-12-01 10:20:53 19919:19919 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:01.9811049Z STAGE:2022-12-01 10:20:53 19917:19917 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:01.9811288Z STAGE:2022-12-01 10:20:53 19918:19918 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:01.9811688Z STAGE:2022-12-01 10:20:53 19919:19919 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-12-01 10:20:53 19917:19917 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:01.9811696Z 
2022-12-01T10:31:01.9811939Z STAGE:2022-12-01 10:20:53 19918:19918 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:01.9812338Z STAGE:2022-12-01 10:20:53 19919:19919 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 10:20:53 19917:19917 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:01.9812353Z 
2022-12-01T10:31:01.9812578Z STAGE:2022-12-01 10:20:53 19918:19918 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:01.9812974Z STAGE:2022-12-01 10:20:53 19919:19919 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-12-01 10:20:53 19917:19917 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:01.9812979Z 
2022-12-01T10:31:01.9813224Z STAGE:2022-12-01 10:20:53 19918:19918 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:01.9813629Z STAGE:2022-12-01 10:20:53 19919:19919 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 10:20:53 19917:19917 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:01.9813637Z 
2022-12-01T10:31:01.9814196Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1578: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-12-01T10:31:01.9814349Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2022-12-01T10:31:01.9814908Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1578: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-12-01T10:31:01.9815084Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2022-12-01T10:31:01.9815744Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1578: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-12-01T10:31:01.9815891Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2022-12-01T10:31:01.9815959Z ok (1.952s)
2022-12-01T10:31:01.9815964Z 
2022-12-01T10:31:01.9816164Z ----------------------------------------------------------------------
2022-12-01T10:31:01.9816227Z Ran 1 test in 1.953s
2022-12-01T10:31:01.9816232Z 
2022-12-01T10:31:01.9816291Z OK
2022-12-01T10:31:01.9816297Z 
2022-12-01T10:31:01.9816376Z Generating XML reports...
2022-12-01T10:31:01.9816717Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102051.xml
2022-12-01T10:31:01.9816947Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:01.9817227Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:01.9817355Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:01.9817637Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:01.9817771Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:01.9817776Z 
2022-12-01T10:31:01.9817836Z Running tests...
2022-12-01T10:31:01.9818035Z ----------------------------------------------------------------------
2022-12-01T10:31:01.9818278Z   test_all_reduce_coalesced_group_max (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19982
2022-12-01T10:31:01.9818436Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19983
2022-12-01T10:31:01.9818590Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 19984
2022-12-01T10:31:01.9818866Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:01.9818989Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:01.9819266Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:01.9819388Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:01.9819566Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:31:01.9819833Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:01.9819958Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:01.9820235Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:01.9820473Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:01.9820652Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:31:01.9820926Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:01.9821048Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:01.9821314Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:01.9821450Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:01.9821622Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:31:01.9821976Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:01.9822307Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:01.9822597Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:01.9822758Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:31:01.9822922Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:31:01.9823081Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:31:01.9823244Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T10:31:01.9823422Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T10:31:01.9823591Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-12-01T10:31:01.9823884Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:31:01.9824176Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:31:01.9824426Z STAGE:2022-12-01 10:20:56 19983:19983 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:01.9824673Z STAGE:2022-12-01 10:20:56 19984:19984 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:01.9825086Z STAGE:2022-12-01 10:20:56 19984:19984 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 10:20:56 19983:19983 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:01.9825092Z 
2022-12-01T10:31:01.9825488Z STAGE:2022-12-01 10:20:56 19983:19983 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-12-01 10:20:56 19984:19984 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:01.9825496Z 
2022-12-01T10:31:01.9825744Z STAGE:2022-12-01 10:20:56 19984:19984 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:01.9825973Z STAGE:2022-12-01 10:20:56 19983:19983 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:01.9826273Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:31:01.9826833Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1578: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-12-01T10:31:01.9826986Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2022-12-01T10:31:01.9827533Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1578: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-12-01T10:31:01.9827685Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2022-12-01T10:31:01.9827750Z ok (1.955s)
2022-12-01T10:31:01.9827756Z 
2022-12-01T10:31:01.9827956Z ----------------------------------------------------------------------
2022-12-01T10:31:01.9828031Z Ran 1 test in 1.955s
2022-12-01T10:31:01.9828036Z 
2022-12-01T10:31:01.9828096Z OK
2022-12-01T10:31:01.9828101Z 
2022-12-01T10:31:01.9828182Z Generating XML reports...
2022-12-01T10:31:01.9828506Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102054.xml
2022-12-01T10:31:01.9828768Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:01.9829069Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:01.9829193Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:01.9829474Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:01.9829607Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:01.9829612Z 
2022-12-01T10:31:01.9829685Z Running tests...
2022-12-01T10:31:01.9829875Z ----------------------------------------------------------------------
2022-12-01T10:31:01.9830109Z   test_all_reduce_coalesced_group_min (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20043
2022-12-01T10:31:01.9830269Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20044
2022-12-01T10:31:01.9830421Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20045
2022-12-01T10:31:01.9830694Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:01.9830817Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:01.9831097Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:01.9831232Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:01.9831407Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:31:01.9831678Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:01.9831789Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:01.9832072Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:01.9832207Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:01.9832381Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:31:01.9832650Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:01.9832770Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:01.9833045Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:01.9833176Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:01.9833339Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:31:01.9833641Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:01.9833937Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:01.9834226Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:01.9834391Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:31:01.9834554Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:31:01.9834712Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:31:01.9834886Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T10:31:01.9835059Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T10:31:01.9835250Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-12-01T10:31:01.9835569Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:31:01.9835859Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:31:01.9836108Z STAGE:2022-12-01 10:20:59 20044:20044 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:01.9836400Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:31:01.9836647Z STAGE:2022-12-01 10:20:59 20045:20045 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:01.9836892Z STAGE:2022-12-01 10:20:59 20045:20045 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:01.9837136Z STAGE:2022-12-01 10:20:59 20044:20044 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:01.9837376Z STAGE:2022-12-01 10:20:59 20045:20045 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:01.9837614Z STAGE:2022-12-01 10:20:59 20044:20044 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:01.9837846Z STAGE:2022-12-01 10:20:59 20045:20045 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:01.9838089Z STAGE:2022-12-01 10:20:59 20044:20044 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:01.9838653Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1578: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-12-01T10:31:01.9838805Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2022-12-01T10:31:01.9839350Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1578: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-12-01T10:31:01.9839501Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2022-12-01T10:31:01.9839568Z ok (2.054s)
2022-12-01T10:31:01.9839573Z 
2022-12-01T10:31:01.9839770Z ----------------------------------------------------------------------
2022-12-01T10:31:01.9839843Z Ran 1 test in 2.054s
2022-12-01T10:31:01.9839848Z 
2022-12-01T10:31:01.9839907Z OK
2022-12-01T10:31:01.9839912Z 
2022-12-01T10:31:01.9839983Z Generating XML reports...
2022-12-01T10:31:01.9840367Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102058.xml
2022-12-01T10:31:01.9840602Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:01.9840872Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:01.9840997Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:01.9841275Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:01.9841410Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:01.9841415Z 
2022-12-01T10:31:01.9841484Z Running tests...
2022-12-01T10:31:01.9841665Z ----------------------------------------------------------------------
2022-12-01T10:31:01.9841920Z   test_all_reduce_coalesced_group_product (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20104
2022-12-01T10:31:01.9842077Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20105
2022-12-01T10:31:01.9842261Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20106
2022-12-01T10:31:01.9842559Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:01.9842681Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:01.9842961Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:01.9843094Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:01.9843271Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:31:01.9843529Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:01.9843651Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:01.9843932Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:01.9844068Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:01.9844245Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:31:01.9844512Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:01.9844633Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:01.9844910Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:01.9845031Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:01.9845204Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:31:01.9845504Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:01.9845799Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:01.9846088Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:01.9846257Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:31:01.9846416Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:31:01.9846592Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T10:31:01.9846751Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:31:01.9846907Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-12-01T10:31:01.9847079Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T10:31:01.9847369Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:31:01.9847660Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:31:01.9847907Z STAGE:2022-12-01 10:21:03 20105:20105 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:01.9848198Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:31:01.9848445Z STAGE:2022-12-01 10:21:03 20106:20106 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:01.9848856Z STAGE:2022-12-01 10:21:03 20105:20105 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 10:21:03 20106:20106 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:01.9848862Z 
2022-12-01T10:31:01.9849284Z STAGE:2022-12-01 10:21:03 20105:20105 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-12-01 10:21:03 20106:20106 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:01.9849315Z 
2022-12-01T10:31:01.9849561Z STAGE:2022-12-01 10:21:03 20105:20105 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:01.9849802Z STAGE:2022-12-01 10:21:03 20106:20106 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:01.9850367Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1578: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-12-01T10:31:01.9850506Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2022-12-01T10:31:01.9851053Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1578: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-12-01T10:31:01.9851205Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2022-12-01T10:31:01.9851271Z ok (1.855s)
2022-12-01T10:31:01.9851276Z 
2022-12-01T10:31:01.9851473Z ----------------------------------------------------------------------
2022-12-01T10:31:01.9851548Z Ran 1 test in 1.855s
2022-12-01T10:31:01.9851553Z 
2022-12-01T10:31:01.9851611Z OK
2022-12-01T10:31:01.9851616Z 
2022-12-01T10:31:01.9851698Z Generating XML reports...
2022-12-01T10:31:01.9852034Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102101.xml
2022-12-01T10:31:01.9852254Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:01.9852528Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:01.9852650Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:01.9852929Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:01.9853060Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:01.9853065Z 
2022-12-01T10:31:01.9853137Z Running tests...
2022-12-01T10:31:01.9853327Z ----------------------------------------------------------------------
2022-12-01T10:31:01.9853576Z   test_all_reduce_coalesced_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20165
2022-12-01T10:31:01.9853729Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20166
2022-12-01T10:31:01.9853869Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20167
2022-12-01T10:31:01.9854143Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:01.9854268Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:01.9854549Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:01.9854681Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:01.9854855Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:31:01.9855124Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:01.9855316Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:01.9855590Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:01.9855774Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:01.9855979Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:31:01.9856248Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:01.9856371Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:01.9856647Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:01.9856782Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:01.9856958Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:31:01.9857257Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:01.9857543Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:01.9857835Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:01.9858001Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:31:01.9858166Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:31:01.9858341Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T10:31:01.9858501Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:31:01.9858665Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T10:31:01.9858842Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-12-01T10:31:01.9859132Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:31:01.9859508Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:31:01.9859756Z STAGE:2022-12-01 10:21:06 20166:20166 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:01.9860001Z STAGE:2022-12-01 10:21:06 20167:20167 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:01.9860499Z STAGE:2022-12-01 10:21:06 20166:20166 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 10:21:06 20167:20167 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:01.9860507Z 
2022-12-01T10:31:01.9860910Z STAGE:2022-12-01 10:21:06 20166:20166 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-12-01 10:21:06 20167:20167 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:01.9860918Z 
2022-12-01T10:31:01.9861330Z STAGE:2022-12-01 10:21:06 20167:20167 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 10:21:06 20166:20166 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:01.9861338Z 
2022-12-01T10:31:01.9861639Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:31:01.9862206Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1578: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-12-01T10:31:01.9862362Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2022-12-01T10:31:01.9862971Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1578: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-12-01T10:31:01.9863157Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2022-12-01T10:31:01.9863226Z ok (1.853s)
2022-12-01T10:31:01.9863232Z 
2022-12-01T10:31:01.9863433Z ----------------------------------------------------------------------
2022-12-01T10:31:01.9863495Z Ran 1 test in 1.853s
2022-12-01T10:31:01.9863500Z 
2022-12-01T10:31:01.9863563Z OK
2022-12-01T10:31:01.9863568Z 
2022-12-01T10:31:01.9863651Z Generating XML reports...
2022-12-01T10:31:01.9863996Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102105.xml
2022-12-01T10:31:01.9864231Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:01.9864510Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:01.9864638Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:01.9864923Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:01.9865046Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:01.9865064Z 
2022-12-01T10:31:01.9865124Z Running tests...
2022-12-01T10:31:01.9865319Z ----------------------------------------------------------------------
2022-12-01T10:31:01.9865562Z   test_all_reduce_coalesced_max (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20226
2022-12-01T10:31:01.9865721Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20227
2022-12-01T10:31:01.9865878Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20228
2022-12-01T10:31:01.9866152Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:01.9866279Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:01.9866560Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:01.9866683Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:01.9866863Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:31:01.9867133Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:01.9867256Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:01.9867534Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:01.9867668Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:01.9867846Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:31:01.9868116Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:01.9868240Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:01.9868506Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:01.9868641Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:01.9868817Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:31:01.9869116Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:01.9869440Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:01.9869735Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:01.9869932Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:31:01.9870095Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:31:01.9870255Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:31:01.9870489Z STAGE:2022-12-01 10:21:09 20226:20226 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:01.9870731Z STAGE:2022-12-01 10:21:09 20228:20228 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:01.9870973Z STAGE:2022-12-01 10:21:09 20227:20227 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:01.9871553Z STAGE:2022-12-01 10:21:09 20226:20226 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 10:21:09 20227:20227 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 10:21:09 20228:20228 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:01.9871561Z 
2022-12-01T10:31:01.9871565Z 
2022-12-01T10:31:01.9871806Z STAGE:2022-12-01 10:21:09 20226:20226 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:01.9872203Z STAGE:2022-12-01 10:21:09 20227:20227 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-12-01 10:21:09 20228:20228 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:01.9872209Z 
2022-12-01T10:31:01.9872457Z STAGE:2022-12-01 10:21:09 20228:20228 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:01.9872701Z STAGE:2022-12-01 10:21:09 20227:20227 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:01.9872945Z STAGE:2022-12-01 10:21:09 20226:20226 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:01.9873183Z STAGE:2022-12-01 10:21:09 20228:20228 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:01.9873418Z STAGE:2022-12-01 10:21:09 20226:20226 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:01.9873638Z STAGE:2022-12-01 10:21:09 20227:20227 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:01.9874046Z STAGE:2022-12-01 10:21:09 20227:20227 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 10:21:09 20228:20228 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:01.9874051Z 
2022-12-01T10:31:01.9874290Z STAGE:2022-12-01 10:21:09 20226:20226 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:01.9874858Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1578: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-12-01T10:31:01.9875013Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2022-12-01T10:31:01.9875563Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1578: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-12-01T10:31:01.9875709Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2022-12-01T10:31:01.9876262Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1578: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-12-01T10:31:01.9876438Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2022-12-01T10:31:01.9876508Z ok (1.853s)
2022-12-01T10:31:01.9876539Z 
2022-12-01T10:31:01.9876736Z ----------------------------------------------------------------------
2022-12-01T10:31:01.9876798Z Ran 1 test in 1.853s
2022-12-01T10:31:01.9876815Z 
2022-12-01T10:31:01.9876862Z OK
2022-12-01T10:31:01.9876866Z 
2022-12-01T10:31:01.9876949Z Generating XML reports...
2022-12-01T10:31:01.9877294Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102108.xml
2022-12-01T10:31:01.9877524Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:01.9877799Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:01.9877922Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:01.9878206Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:01.9878345Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:01.9878350Z 
2022-12-01T10:31:01.9878409Z Running tests...
2022-12-01T10:31:01.9878607Z ----------------------------------------------------------------------
2022-12-01T10:31:01.9878871Z   test_all_reduce_coalesced_max_complex_unsupported (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20282
2022-12-01T10:31:01.9879029Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20283
2022-12-01T10:31:01.9879184Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20284
2022-12-01T10:31:01.9879457Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:01.9879581Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:01.9879864Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:01.9879999Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:01.9880164Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:31:01.9880439Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:01.9880560Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:01.9880838Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:01.9880972Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:01.9881148Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:31:01.9881422Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:01.9881548Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:01.9881815Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:01.9881950Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:01.9882128Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:31:01.9882425Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:01.9882722Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:01.9883013Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:01.9883210Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:31:01.9883405Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:31:01.9883567Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:31:01.9884130Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1578: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-12-01T10:31:01.9884269Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2022-12-01T10:31:01.9884827Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1578: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-12-01T10:31:01.9884979Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2022-12-01T10:31:01.9885531Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1578: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-12-01T10:31:01.9885681Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2022-12-01T10:31:01.9885749Z ok (1.857s)
2022-12-01T10:31:01.9885754Z 
2022-12-01T10:31:01.9885952Z ----------------------------------------------------------------------
2022-12-01T10:31:01.9886028Z Ran 1 test in 1.857s
2022-12-01T10:31:01.9886033Z 
2022-12-01T10:31:01.9886094Z OK
2022-12-01T10:31:01.9886098Z 
2022-12-01T10:31:01.9886182Z Generating XML reports...
2022-12-01T10:31:01.9886512Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102111.xml
2022-12-01T10:31:01.9886745Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:01.9887019Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:01.9887142Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:01.9887421Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:01.9887556Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:01.9887561Z 
2022-12-01T10:31:01.9887633Z Running tests...
2022-12-01T10:31:01.9887830Z ----------------------------------------------------------------------
2022-12-01T10:31:01.9888057Z   test_all_reduce_coalesced_min (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20335
2022-12-01T10:31:01.9888215Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20336
2022-12-01T10:31:01.9888372Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20337
2022-12-01T10:31:01.9888644Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:01.9888769Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:01.9889049Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:01.9889184Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:01.9889364Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:31:01.9889638Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:01.9889779Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:01.9890083Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:01.9890217Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:01.9890396Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:31:01.9890666Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:01.9890789Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:01.9891069Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:01.9891202Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:01.9891379Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:31:01.9891665Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:01.9891964Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:01.9892258Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:01.9892421Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:31:01.9892586Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:31:01.9892747Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:31:01.9892995Z STAGE:2022-12-01 10:21:16 20337:20337 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:01.9893238Z STAGE:2022-12-01 10:21:16 20335:20335 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:01.9893482Z STAGE:2022-12-01 10:21:16 20336:20336 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:01.9893880Z STAGE:2022-12-01 10:21:16 20337:20337 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 10:21:16 20336:20336 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:01.9894129Z STAGE:2022-12-01 10:21:16 20335:20335 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:01.9894134Z 
2022-12-01T10:31:01.9894681Z STAGE:2022-12-01 10:21:16 20336:20336 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-12-01 10:21:16 20335:20335 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-12-01 10:21:16 20337:20337 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:01.9894687Z 
2022-12-01T10:31:01.9894691Z 
2022-12-01T10:31:01.9894938Z STAGE:2022-12-01 10:21:16 20335:20335 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:01.9895427Z STAGE:2022-12-01 10:21:16 20337:20337 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 10:21:16 20336:20336 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:01.9895433Z 
2022-12-01T10:31:01.9895677Z STAGE:2022-12-01 10:21:16 20335:20335 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:01.9895958Z STAGE:2022-12-01 10:21:16 20337:20337 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:01.9896200Z STAGE:2022-12-01 10:21:16 20336:20336 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:01.9896447Z STAGE:2022-12-01 10:21:16 20335:20335 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:01.9896895Z STAGE:2022-12-01 10:21:16 20336:20336 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 10:21:16 20337:20337 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:01.9896927Z 
2022-12-01T10:31:01.9897497Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1578: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-12-01T10:31:01.9897652Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2022-12-01T10:31:01.9898207Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1578: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-12-01T10:31:01.9898358Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2022-12-01T10:31:01.9898900Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1578: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-12-01T10:31:01.9899051Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2022-12-01T10:31:01.9899122Z ok (1.861s)
2022-12-01T10:31:01.9899127Z 
2022-12-01T10:31:01.9899323Z ----------------------------------------------------------------------
2022-12-01T10:31:01.9899398Z Ran 1 test in 1.862s
2022-12-01T10:31:01.9899403Z 
2022-12-01T10:31:01.9899464Z OK
2022-12-01T10:31:01.9899468Z 
2022-12-01T10:31:01.9899554Z Generating XML reports...
2022-12-01T10:31:01.9899894Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102115.xml
2022-12-01T10:31:01.9900126Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:01.9900498Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:01.9900626Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:01.9900910Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:01.9901047Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:01.9901052Z 
2022-12-01T10:31:01.9901126Z Running tests...
2022-12-01T10:31:01.9901323Z ----------------------------------------------------------------------
2022-12-01T10:31:01.9901576Z   test_all_reduce_coalesced_product (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20391
2022-12-01T10:31:01.9901736Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20392
2022-12-01T10:31:01.9901894Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20393
2022-12-01T10:31:01.9902153Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:01.9902281Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:01.9902562Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:01.9902699Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:01.9902880Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:31:01.9903181Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:01.9903452Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:01.9903577Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:01.9903911Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:01.9904102Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:01.9904277Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:31:01.9904552Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:01.9904676Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:01.9904957Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:01.9905093Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:01.9905269Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:31:01.9905571Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:01.9905855Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:01.9906021Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:31:01.9906182Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:31:01.9906344Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:31:01.9906594Z STAGE:2022-12-01 10:21:19 20391:20391 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:01.9906835Z STAGE:2022-12-01 10:21:19 20393:20393 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:01.9907070Z STAGE:2022-12-01 10:21:19 20392:20392 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:01.9907484Z STAGE:2022-12-01 10:21:19 20391:20391 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 10:21:19 20392:20392 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:01.9907734Z STAGE:2022-12-01 10:21:19 20393:20393 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:01.9907740Z 
2022-12-01T10:31:01.9908136Z STAGE:2022-12-01 10:21:19 20392:20392 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-12-01 10:21:19 20391:20391 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:01.9908361Z STAGE:2022-12-01 10:21:19 20393:20393 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:01.9908381Z 
2022-12-01T10:31:01.9908615Z STAGE:2022-12-01 10:21:19 20392:20392 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:01.9909028Z STAGE:2022-12-01 10:21:19 20391:20391 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 10:21:19 20393:20393 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:01.9909035Z 
2022-12-01T10:31:01.9909278Z STAGE:2022-12-01 10:21:19 20392:20392 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:01.9909523Z STAGE:2022-12-01 10:21:19 20391:20391 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:01.9909761Z STAGE:2022-12-01 10:21:19 20393:20393 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:01.9910008Z STAGE:2022-12-01 10:21:19 20393:20393 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:01.9910250Z STAGE:2022-12-01 10:21:19 20391:20391 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:01.9910486Z STAGE:2022-12-01 10:21:19 20392:20392 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:01.9911093Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1578: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-12-01T10:31:01.9911278Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2022-12-01T10:31:01.9911828Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1578: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-12-01T10:31:01.9911977Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2022-12-01T10:31:01.9912529Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1578: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-12-01T10:31:01.9912680Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2022-12-01T10:31:01.9912751Z ok (1.855s)
2022-12-01T10:31:01.9912756Z 
2022-12-01T10:31:01.9912957Z ----------------------------------------------------------------------
2022-12-01T10:31:01.9913032Z Ran 1 test in 1.855s
2022-12-01T10:31:01.9913037Z 
2022-12-01T10:31:01.9913097Z OK
2022-12-01T10:31:01.9913102Z 
2022-12-01T10:31:01.9913185Z Generating XML reports...
2022-12-01T10:31:01.9913569Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102118.xml
2022-12-01T10:31:01.9913933Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:01.9914378Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:01.9914568Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:01.9914962Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:01.9915105Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:01.9915110Z 
2022-12-01T10:31:01.9915184Z Running tests...
2022-12-01T10:31:01.9915387Z ----------------------------------------------------------------------
2022-12-01T10:31:01.9915628Z   test_all_reduce_coalesced_sum (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20447
2022-12-01T10:31:01.9915785Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20448
2022-12-01T10:31:01.9915927Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20449
2022-12-01T10:31:01.9916205Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:01.9916332Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:01.9916614Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:01.9916753Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:01.9916934Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:31:01.9917210Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:01.9917334Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:01.9917599Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:01.9917735Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:01.9917914Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:31:01.9918238Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:01.9918396Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:01.9918674Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:01.9918809Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:01.9918987Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:31:01.9919288Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:01.9919572Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:01.9919866Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:01.9920041Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:31:01.9920205Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:31:01.9920456Z STAGE:2022-12-01 10:21:23 20448:20448 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:01.9920698Z STAGE:2022-12-01 10:21:23 20449:20449 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:01.9920864Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:31:01.9921103Z STAGE:2022-12-01 10:21:23 20447:20447 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:01.9921354Z STAGE:2022-12-01 10:21:23 20447:20447 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:01.9921757Z STAGE:2022-12-01 10:21:23 20448:20448 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 10:21:23 20449:20449 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:01.9921779Z 
2022-12-01T10:31:01.9922009Z STAGE:2022-12-01 10:21:23 20447:20447 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:01.9922404Z STAGE:2022-12-01 10:21:23 20449:20449 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-12-01 10:21:23 20448:20448 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:01.9922410Z 
2022-12-01T10:31:01.9922817Z STAGE:2022-12-01 10:21:23 20449:20449 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 10:21:23 20447:20447 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:01.9923065Z STAGE:2022-12-01 10:21:23 20448:20448 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:01.9923070Z 
2022-12-01T10:31:01.9923314Z STAGE:2022-12-01 10:21:23 20447:20447 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:01.9923552Z STAGE:2022-12-01 10:21:23 20448:20448 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:01.9923787Z STAGE:2022-12-01 10:21:23 20449:20449 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:01.9924034Z STAGE:2022-12-01 10:21:23 20448:20448 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:01.9924275Z STAGE:2022-12-01 10:21:23 20447:20447 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:01.9924526Z STAGE:2022-12-01 10:21:23 20449:20449 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:01.9925095Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1578: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-12-01T10:31:01.9925265Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2022-12-01T10:31:01.9925857Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1578: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-12-01T10:31:01.9926004Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2022-12-01T10:31:01.9926563Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1578: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-12-01T10:31:01.9926708Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2022-12-01T10:31:01.9926776Z ok (1.954s)
2022-12-01T10:31:01.9926781Z 
2022-12-01T10:31:01.9926983Z ----------------------------------------------------------------------
2022-12-01T10:31:01.9927061Z Ran 1 test in 1.954s
2022-12-01T10:31:01.9927066Z 
2022-12-01T10:31:01.9927127Z OK
2022-12-01T10:31:01.9927131Z 
2022-12-01T10:31:01.9927216Z Generating XML reports...
2022-12-01T10:31:01.9927546Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102121.xml
2022-12-01T10:31:01.9927779Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:01.9928052Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:01.9928179Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:01.9928460Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:01.9928600Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:01.9928605Z 
2022-12-01T10:31:01.9928681Z Running tests...
2022-12-01T10:31:01.9928879Z ----------------------------------------------------------------------
2022-12-01T10:31:01.9929119Z   test_all_reduce_complex_unsupported_ops (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20503
2022-12-01T10:31:01.9929277Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20504
2022-12-01T10:31:01.9929431Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20505
2022-12-01T10:31:01.9929700Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:01.9929826Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:01.9930108Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:01.9930247Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:01.9930428Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:31:01.9930698Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:01.9930807Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:01.9931088Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:01.9931222Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:01.9931402Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:31:01.9931701Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:01.9932002Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:01.9932151Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:01.9932431Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:01.9932564Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:01.9932726Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:31:01.9933028Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:01.9933322Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:01.9933489Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:31:01.9933655Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:31:01.9933821Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:31:01.9933888Z ok (1.758s)
2022-12-01T10:31:01.9933893Z 
2022-12-01T10:31:01.9934090Z ----------------------------------------------------------------------
2022-12-01T10:31:01.9934151Z Ran 1 test in 1.758s
2022-12-01T10:31:01.9934169Z 
2022-12-01T10:31:01.9934217Z OK
2022-12-01T10:31:01.9934222Z 
2022-12-01T10:31:01.9934305Z Generating XML reports...
2022-12-01T10:31:01.9934644Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102125.xml
2022-12-01T10:31:01.9934874Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:01.9935147Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:01.9935375Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:01.9935662Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:01.9935798Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:01.9935804Z 
2022-12-01T10:31:01.9935864Z Running tests...
2022-12-01T10:31:01.9936060Z ----------------------------------------------------------------------
2022-12-01T10:31:01.9936299Z   test_all_reduce_full_group_max (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20556
2022-12-01T10:31:01.9936455Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20557
2022-12-01T10:31:01.9936611Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20558
2022-12-01T10:31:01.9936884Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:01.9937010Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:01.9937292Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:01.9937429Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:01.9937593Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:31:01.9937864Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:01.9937987Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:01.9938262Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:01.9938397Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:01.9938613Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:31:01.9938908Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:01.9939031Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:01.9939294Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:01.9939427Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:01.9939602Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:31:01.9939897Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:01.9940194Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:01.9940635Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:01.9940808Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:31:01.9940973Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:31:01.9941138Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:31:01.9941301Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T10:31:01.9941475Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-12-01T10:31:01.9941649Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T10:31:01.9941944Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:31:01.9942238Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:31:01.9942529Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:31:01.9942779Z STAGE:2022-12-01 10:21:29 20556:20556 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:01.9943022Z STAGE:2022-12-01 10:21:29 20558:20558 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:01.9943258Z STAGE:2022-12-01 10:21:29 20557:20557 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:01.9943493Z STAGE:2022-12-01 10:21:29 20556:20556 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:01.9943905Z STAGE:2022-12-01 10:21:29 20557:20557 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 10:21:29 20558:20558 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:01.9943912Z 
2022-12-01T10:31:01.9944157Z STAGE:2022-12-01 10:21:29 20556:20556 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:01.9944394Z STAGE:2022-12-01 10:21:29 20557:20557 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:01.9944628Z STAGE:2022-12-01 10:21:29 20558:20558 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:01.9944873Z STAGE:2022-12-01 10:21:29 20556:20556 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:01.9945284Z STAGE:2022-12-01 10:21:29 20558:20558 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 10:21:29 20557:20557 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:01.9945289Z 
2022-12-01T10:31:01.9945530Z STAGE:2022-12-01 10:21:29 20557:20557 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:01.9945844Z STAGE:2022-12-01 10:21:29 20556:20556 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:01.9946114Z STAGE:2022-12-01 10:21:29 20558:20558 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:01.9946365Z STAGE:2022-12-01 10:21:29 20557:20557 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:01.9946595Z STAGE:2022-12-01 10:21:29 20556:20556 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:01.9946839Z STAGE:2022-12-01 10:21:29 20558:20558 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:01.9946908Z ok (1.855s)
2022-12-01T10:31:01.9946913Z 
2022-12-01T10:31:01.9947113Z ----------------------------------------------------------------------
2022-12-01T10:31:01.9947193Z Ran 1 test in 1.856s
2022-12-01T10:31:01.9947198Z 
2022-12-01T10:31:01.9947260Z OK
2022-12-01T10:31:01.9947264Z 
2022-12-01T10:31:01.9947349Z Generating XML reports...
2022-12-01T10:31:01.9947726Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102128.xml
2022-12-01T10:31:01.9947949Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:01.9948223Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:01.9948348Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:01.9948632Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:01.9948770Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:01.9948775Z 
2022-12-01T10:31:01.9948849Z Running tests...
2022-12-01T10:31:01.9949047Z ----------------------------------------------------------------------
2022-12-01T10:31:01.9949286Z   test_all_reduce_full_group_min (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20621
2022-12-01T10:31:01.9949443Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20622
2022-12-01T10:31:01.9949587Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20623
2022-12-01T10:31:01.9949872Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:01.9950043Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:01.9950377Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:01.9950514Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:01.9950690Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:31:01.9950964Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:01.9951089Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:01.9951371Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:01.9951494Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:01.9951670Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:31:01.9951939Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:01.9952062Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:01.9952338Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:01.9952472Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:01.9952706Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:31:01.9953009Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:01.9953318Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:01.9953613Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:01.9953777Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:31:01.9953941Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:31:01.9954103Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:31:01.9954278Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T10:31:01.9954457Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-12-01T10:31:01.9954631Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T10:31:01.9954927Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:31:01.9955209Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:31:01.9955502Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:31:01.9955755Z STAGE:2022-12-01 10:21:33 20621:20621 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:01.9956159Z STAGE:2022-12-01 10:21:33 20622:20622 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-12-01 10:21:33 20623:20623 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:01.9956167Z 
2022-12-01T10:31:01.9956418Z STAGE:2022-12-01 10:21:33 20623:20623 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:01.9956669Z STAGE:2022-12-01 10:21:33 20621:20621 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:01.9956913Z STAGE:2022-12-01 10:21:33 20622:20622 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:01.9957157Z STAGE:2022-12-01 10:21:33 20623:20623 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:01.9957392Z STAGE:2022-12-01 10:21:33 20621:20621 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:01.9957626Z STAGE:2022-12-01 10:21:33 20622:20622 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:01.9957857Z STAGE:2022-12-01 10:21:33 20623:20623 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:01.9958101Z STAGE:2022-12-01 10:21:33 20621:20621 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:01.9958341Z STAGE:2022-12-01 10:21:33 20622:20622 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:01.9958581Z STAGE:2022-12-01 10:21:33 20623:20623 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:01.9958818Z STAGE:2022-12-01 10:21:33 20621:20621 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:01.9959059Z STAGE:2022-12-01 10:21:33 20622:20622 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:01.9959468Z STAGE:2022-12-01 10:21:33 20623:20623 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 10:21:33 20622:20622 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:01.9959474Z 
2022-12-01T10:31:01.9959718Z STAGE:2022-12-01 10:21:33 20621:20621 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:01.9959788Z ok (1.967s)
2022-12-01T10:31:01.9959793Z 
2022-12-01T10:31:01.9960024Z ----------------------------------------------------------------------
2022-12-01T10:31:01.9960114Z Ran 1 test in 1.967s
2022-12-01T10:31:01.9960119Z 
2022-12-01T10:31:01.9960181Z OK
2022-12-01T10:31:01.9960186Z 
2022-12-01T10:31:01.9960273Z Generating XML reports...
2022-12-01T10:31:01.9960644Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102131.xml
2022-12-01T10:31:01.9960987Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:01.9961311Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:01.9961437Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:01.9961719Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:01.9961844Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:01.9961867Z 
2022-12-01T10:31:01.9961927Z Running tests...
2022-12-01T10:31:01.9962123Z ----------------------------------------------------------------------
2022-12-01T10:31:01.9962371Z   test_all_reduce_full_group_product (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20686
2022-12-01T10:31:01.9962531Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20687
2022-12-01T10:31:01.9962687Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20688
2022-12-01T10:31:01.9962964Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:01.9963088Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:01.9963371Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:01.9963497Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:01.9963680Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:31:01.9963950Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:01.9964073Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:01.9964352Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:01.9964488Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:01.9964667Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:31:01.9964965Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:01.9965234Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:01.9965346Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:01.9965628Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:01.9965761Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:01.9965934Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:31:01.9966228Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:01.9966523Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:01.9966690Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:31:01.9966894Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:31:01.9967070Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:31:01.9967247Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T10:31:01.9967419Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-12-01T10:31:01.9967591Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T10:31:01.9967884Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:31:01.9968176Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:31:01.9968466Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:31:01.9968717Z STAGE:2022-12-01 10:21:36 20686:20686 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:01.9969124Z STAGE:2022-12-01 10:21:36 20687:20687 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-12-01 10:21:36 20688:20688 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:01.9969129Z 
2022-12-01T10:31:01.9969377Z STAGE:2022-12-01 10:21:36 20686:20686 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:01.9969606Z STAGE:2022-12-01 10:21:36 20688:20688 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:01.9969853Z STAGE:2022-12-01 10:21:36 20687:20687 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:01.9970099Z STAGE:2022-12-01 10:21:36 20686:20686 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:01.9970340Z STAGE:2022-12-01 10:21:36 20688:20688 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:01.9970577Z STAGE:2022-12-01 10:21:36 20687:20687 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:01.9970821Z STAGE:2022-12-01 10:21:36 20688:20688 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:01.9971068Z STAGE:2022-12-01 10:21:36 20687:20687 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:01.9971308Z STAGE:2022-12-01 10:21:36 20686:20686 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:01.9971554Z STAGE:2022-12-01 10:21:36 20688:20688 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:01.9971956Z STAGE:2022-12-01 10:21:36 20687:20687 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-12-01 10:21:36 20686:20686 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:01.9971962Z 
2022-12-01T10:31:01.9972198Z STAGE:2022-12-01 10:21:36 20688:20688 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:01.9972610Z STAGE:2022-12-01 10:21:36 20687:20687 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 10:21:36 20686:20686 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:01.9972615Z 
2022-12-01T10:31:01.9972682Z ok (1.851s)
2022-12-01T10:31:01.9972687Z 
2022-12-01T10:31:01.9972885Z ----------------------------------------------------------------------
2022-12-01T10:31:01.9972960Z Ran 1 test in 1.851s
2022-12-01T10:31:01.9972965Z 
2022-12-01T10:31:01.9973024Z OK
2022-12-01T10:31:01.9973029Z 
2022-12-01T10:31:01.9973112Z Generating XML reports...
2022-12-01T10:31:01.9973455Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102135.xml
2022-12-01T10:31:01.9973687Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:01.9973977Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:01.9974132Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:01.9974410Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:01.9974546Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:01.9974551Z 
2022-12-01T10:31:01.9974624Z Running tests...
2022-12-01T10:31:01.9974818Z ----------------------------------------------------------------------
2022-12-01T10:31:01.9975056Z   test_all_reduce_full_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20751
2022-12-01T10:31:01.9975214Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20752
2022-12-01T10:31:01.9975447Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20753
2022-12-01T10:31:01.9975713Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:01.9975839Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:01.9976122Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:01.9976257Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:01.9976436Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:31:01.9976736Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:01.9977010Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:01.9977136Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:01.9977420Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:01.9977543Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:01.9977723Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:31:01.9977995Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:01.9978120Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:01.9978398Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:01.9978532Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:01.9978708Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:31:01.9979010Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:01.9979310Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:01.9979463Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:31:01.9979624Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:31:01.9979782Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:31:01.9979958Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T10:31:01.9980131Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-12-01T10:31:01.9980303Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T10:31:01.9980795Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:31:01.9981129Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:31:01.9981425Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:31:01.9981660Z STAGE:2022-12-01 10:21:40 20751:20751 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:01.9981900Z STAGE:2022-12-01 10:21:40 20752:20752 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:01.9982143Z STAGE:2022-12-01 10:21:40 20753:20753 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:01.9982393Z STAGE:2022-12-01 10:21:40 20751:20751 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:01.9982641Z STAGE:2022-12-01 10:21:40 20753:20753 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:01.9982885Z STAGE:2022-12-01 10:21:40 20752:20752 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:01.9983283Z STAGE:2022-12-01 10:21:40 20751:20751 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-12-01 10:21:40 20752:20752 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:01.9983291Z 
2022-12-01T10:31:01.9983534Z STAGE:2022-12-01 10:21:40 20753:20753 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:01.9983780Z STAGE:2022-12-01 10:21:40 20751:20751 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:01.9984008Z STAGE:2022-12-01 10:21:40 20753:20753 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:01.9984245Z STAGE:2022-12-01 10:21:40 20752:20752 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:01.9984487Z STAGE:2022-12-01 10:21:40 20751:20751 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:01.9984731Z STAGE:2022-12-01 10:21:40 20753:20753 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:01.9984969Z STAGE:2022-12-01 10:21:40 20752:20752 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:01.9985214Z STAGE:2022-12-01 10:21:40 20753:20753 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:01.9985457Z STAGE:2022-12-01 10:21:40 20752:20752 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:01.9985693Z STAGE:2022-12-01 10:21:40 20751:20751 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:01.9985760Z ok (1.953s)
2022-12-01T10:31:01.9985766Z 
2022-12-01T10:31:01.9985951Z ----------------------------------------------------------------------
2022-12-01T10:31:01.9986025Z Ran 1 test in 1.954s
2022-12-01T10:31:01.9986031Z 
2022-12-01T10:31:01.9986091Z OK
2022-12-01T10:31:01.9986096Z 
2022-12-01T10:31:01.9986182Z Generating XML reports...
2022-12-01T10:31:01.9986525Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102138.xml
2022-12-01T10:31:01.9986756Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:01.9987034Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:01.9987159Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:01.9987439Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:01.9987561Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:01.9987566Z 
2022-12-01T10:31:01.9987640Z Running tests...
2022-12-01T10:31:01.9987839Z ----------------------------------------------------------------------
2022-12-01T10:31:01.9988103Z   test_all_reduce_group_max (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20816
2022-12-01T10:31:01.9988291Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20817
2022-12-01T10:31:01.9988444Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20818
2022-12-01T10:31:01.9988717Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:01.9988840Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:01.9989107Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:01.9989243Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:01.9989421Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:31:01.9989693Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:01.9989819Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:01.9990099Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:01.9990235Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:01.9990413Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:31:01.9990680Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:01.9990791Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:01.9991068Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:01.9991204Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:01.9991378Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:31:01.9991680Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:01.9991977Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:01.9992269Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:01.9992433Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:31:01.9992598Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:31:01.9992759Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-12-01T10:31:01.9992935Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T10:31:01.9993100Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:31:01.9993271Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T10:31:01.9993561Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:31:01.9993855Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:31:01.9994144Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:31:01.9994391Z STAGE:2022-12-01 10:21:43 20817:20817 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:01.9994661Z STAGE:2022-12-01 10:21:43 20818:20818 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:01.9994899Z STAGE:2022-12-01 10:21:43 20818:20818 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:01.9995169Z STAGE:2022-12-01 10:21:43 20817:20817 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:01.9995407Z STAGE:2022-12-01 10:21:43 20818:20818 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:01.9995650Z STAGE:2022-12-01 10:21:43 20817:20817 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:01.9995895Z STAGE:2022-12-01 10:21:43 20818:20818 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:01.9996137Z STAGE:2022-12-01 10:21:43 20817:20817 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:01.9996204Z ok (2.057s)
2022-12-01T10:31:01.9996209Z 
2022-12-01T10:31:01.9996405Z ----------------------------------------------------------------------
2022-12-01T10:31:01.9996468Z Ran 1 test in 2.057s
2022-12-01T10:31:01.9996486Z 
2022-12-01T10:31:01.9996533Z OK
2022-12-01T10:31:01.9996540Z 
2022-12-01T10:31:01.9996624Z Generating XML reports...
2022-12-01T10:31:01.9996969Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102141.xml
2022-12-01T10:31:01.9997201Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:01.9997478Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:01.9997602Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:01.9997882Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:01.9998017Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:01.9998022Z 
2022-12-01T10:31:01.9998082Z Running tests...
2022-12-01T10:31:01.9998279Z ----------------------------------------------------------------------
2022-12-01T10:31:01.9998517Z   test_all_reduce_group_min (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20877
2022-12-01T10:31:01.9998674Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20878
2022-12-01T10:31:01.9998827Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20879
2022-12-01T10:31:01.9999096Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:01.9999219Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:01.9999494Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:01.9999628Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:01.9999796Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:31:02.0000065Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0000188Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0000464Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0000597Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0000773Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:31:02.0001068Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0001335Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0001486Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0001763Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0001926Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0002102Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:31:02.0002396Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0002690Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0002856Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:31:02.0003020Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:31:02.0003179Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:31:02.0003342Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T10:31:02.0003516Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T10:31:02.0003689Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-12-01T10:31:02.0003981Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:31:02.0004277Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:31:02.0004522Z STAGE:2022-12-01 10:21:46 20878:20878 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0004762Z STAGE:2022-12-01 10:21:46 20879:20879 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0005013Z STAGE:2022-12-01 10:21:46 20878:20878 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.0005258Z STAGE:2022-12-01 10:21:46 20879:20879 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.0005658Z STAGE:2022-12-01 10:21:46 20878:20878 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-12-01 10:21:46 20879:20879 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0005665Z 
2022-12-01T10:31:02.0005899Z STAGE:2022-12-01 10:21:46 20878:20878 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.0006140Z STAGE:2022-12-01 10:21:46 20879:20879 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.0006437Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:31:02.0006505Z ok (1.854s)
2022-12-01T10:31:02.0006510Z 
2022-12-01T10:31:02.0006709Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0006786Z Ran 1 test in 1.854s
2022-12-01T10:31:02.0006791Z 
2022-12-01T10:31:02.0006853Z OK
2022-12-01T10:31:02.0006857Z 
2022-12-01T10:31:02.0006940Z Generating XML reports...
2022-12-01T10:31:02.0007281Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102145.xml
2022-12-01T10:31:02.0007498Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.0007773Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0007896Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0008178Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0008317Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0008355Z 
2022-12-01T10:31:02.0008429Z Running tests...
2022-12-01T10:31:02.0008651Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0008890Z   test_all_reduce_group_product (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20938
2022-12-01T10:31:02.0009034Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20939
2022-12-01T10:31:02.0009189Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 20940
2022-12-01T10:31:02.0009459Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0009582Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0009861Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0009997Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0010176Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:31:02.0010447Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0010570Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0010834Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0010969Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0011144Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:31:02.0011411Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0011537Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0011818Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0011953Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0012126Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:31:02.0012425Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0012707Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0012999Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0013164Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:31:02.0013330Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:31:02.0013507Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-12-01T10:31:02.0013680Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T10:31:02.0013842Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:31:02.0014013Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T10:31:02.0014303Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:31:02.0014583Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:31:02.0014871Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:31:02.0015149Z STAGE:2022-12-01 10:21:50 20939:20939 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0015516Z STAGE:2022-12-01 10:21:50 20940:20940 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0015933Z STAGE:2022-12-01 10:21:50 20940:20940 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 10:21:50 20939:20939 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.0015938Z 
2022-12-01T10:31:02.0016182Z STAGE:2022-12-01 10:21:50 20940:20940 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0016427Z STAGE:2022-12-01 10:21:50 20939:20939 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0016674Z STAGE:2022-12-01 10:21:50 20939:20939 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.0016919Z STAGE:2022-12-01 10:21:50 20940:20940 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.0016976Z ok (1.955s)
2022-12-01T10:31:02.0016998Z 
2022-12-01T10:31:02.0017183Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0017262Z Ran 1 test in 1.955s
2022-12-01T10:31:02.0017266Z 
2022-12-01T10:31:02.0017328Z OK
2022-12-01T10:31:02.0017333Z 
2022-12-01T10:31:02.0017418Z Generating XML reports...
2022-12-01T10:31:02.0017764Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102148.xml
2022-12-01T10:31:02.0018001Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.0018276Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0018401Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0018668Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0018806Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0018813Z 
2022-12-01T10:31:02.0018885Z Running tests...
2022-12-01T10:31:02.0019081Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0019318Z   test_all_reduce_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20999
2022-12-01T10:31:02.0019474Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21000
2022-12-01T10:31:02.0019626Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 21001
2022-12-01T10:31:02.0019897Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0020020Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0020287Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0020553Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0020739Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:31:02.0021019Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0021141Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0021423Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0021559Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0021736Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:31:02.0022045Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0022169Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0022481Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0022620Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0022796Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:31:02.0023094Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0023394Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0023687Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0023856Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:31:02.0024002Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:31:02.0024170Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:31:02.0024345Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T10:31:02.0024522Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T10:31:02.0024692Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-12-01T10:31:02.0024985Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:31:02.0025276Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:31:02.0025523Z STAGE:2022-12-01 10:21:53 21000:21000 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0025767Z STAGE:2022-12-01 10:21:53 21001:21001 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0026169Z STAGE:2022-12-01 10:21:53 21001:21001 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 10:21:53 21000:21000 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.0026188Z 
2022-12-01T10:31:02.0026417Z STAGE:2022-12-01 10:21:53 21000:21000 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0026654Z STAGE:2022-12-01 10:21:53 21001:21001 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0026899Z STAGE:2022-12-01 10:21:53 21000:21000 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.0027144Z STAGE:2022-12-01 10:21:53 21001:21001 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.0027442Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:31:02.0027512Z ok (2.054s)
2022-12-01T10:31:02.0027517Z 
2022-12-01T10:31:02.0027714Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0027789Z Ran 1 test in 2.054s
2022-12-01T10:31:02.0027794Z 
2022-12-01T10:31:02.0027856Z OK
2022-12-01T10:31:02.0027860Z 
2022-12-01T10:31:02.0027933Z Generating XML reports...
2022-12-01T10:31:02.0028273Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102152.xml
2022-12-01T10:31:02.0028503Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.0028778Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0028904Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0029216Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0029381Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0029386Z 
2022-12-01T10:31:02.0029511Z Running tests...
2022-12-01T10:31:02.0029692Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0029925Z   test_all_reduce_max (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21060
2022-12-01T10:31:02.0030080Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21061
2022-12-01T10:31:02.0030236Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 21062
2022-12-01T10:31:02.0030506Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0030630Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0030912Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0031051Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0031229Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:31:02.0031487Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0031610Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0031888Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0032024Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0032202Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:31:02.0032472Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0032598Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0032879Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0032999Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0033176Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:31:02.0033475Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0033771Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0034065Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0034229Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:31:02.0034395Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:31:02.0034642Z STAGE:2022-12-01 10:21:57 21061:21061 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0034888Z STAGE:2022-12-01 10:21:57 21062:21062 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0035036Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:31:02.0035276Z STAGE:2022-12-01 10:21:57 21060:21060 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0035522Z STAGE:2022-12-01 10:21:57 21060:21060 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.0035763Z STAGE:2022-12-01 10:21:57 21062:21062 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.0036041Z STAGE:2022-12-01 10:21:57 21061:21061 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.0036308Z STAGE:2022-12-01 10:21:57 21060:21060 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0036542Z STAGE:2022-12-01 10:21:57 21062:21062 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0036775Z STAGE:2022-12-01 10:21:57 21061:21061 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0037020Z STAGE:2022-12-01 10:21:57 21060:21060 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.0037252Z STAGE:2022-12-01 10:21:57 21061:21061 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.0037492Z STAGE:2022-12-01 10:21:57 21062:21062 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.0037735Z STAGE:2022-12-01 10:21:57 21060:21060 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0037973Z STAGE:2022-12-01 10:21:57 21061:21061 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0038209Z STAGE:2022-12-01 10:21:57 21062:21062 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0038456Z STAGE:2022-12-01 10:21:57 21062:21062 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.0038869Z STAGE:2022-12-01 10:21:57 21060:21060 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 10:21:57 21061:21061 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.0038875Z 
2022-12-01T10:31:02.0038944Z ok (1.954s)
2022-12-01T10:31:02.0038949Z 
2022-12-01T10:31:02.0039147Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0039221Z Ran 1 test in 1.955s
2022-12-01T10:31:02.0039226Z 
2022-12-01T10:31:02.0039275Z OK
2022-12-01T10:31:02.0039279Z 
2022-12-01T10:31:02.0039364Z Generating XML reports...
2022-12-01T10:31:02.0039707Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102155.xml
2022-12-01T10:31:02.0039941Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.0040217Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0040341Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0040621Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0040757Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0040762Z 
2022-12-01T10:31:02.0040822Z Running tests...
2022-12-01T10:31:02.0041016Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0041247Z   test_all_reduce_min (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21116
2022-12-01T10:31:02.0041407Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21117
2022-12-01T10:31:02.0041568Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 21118
2022-12-01T10:31:02.0041843Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0041965Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0042242Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0042377Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0042543Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:31:02.0042849Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0043000Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0043279Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0043416Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0043592Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:31:02.0043890Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0044163Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0044285Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0044553Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0044686Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0044864Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:31:02.0045160Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0045457Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0045623Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:31:02.0045788Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:31:02.0045946Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:31:02.0046194Z STAGE:2022-12-01 10:22:00 21116:21116 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0046421Z STAGE:2022-12-01 10:22:00 21117:21117 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0046659Z STAGE:2022-12-01 10:22:00 21118:21118 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0046905Z STAGE:2022-12-01 10:22:00 21116:21116 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.0047146Z STAGE:2022-12-01 10:22:00 21116:21116 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0047557Z STAGE:2022-12-01 10:22:00 21118:21118 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 10:22:00 21117:21117 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.0047563Z 
2022-12-01T10:31:02.0047966Z STAGE:2022-12-01 10:22:00 21117:21117 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-12-01 10:22:00 21118:21118 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0047972Z 
2022-12-01T10:31:02.0048217Z STAGE:2022-12-01 10:22:00 21116:21116 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.0048625Z STAGE:2022-12-01 10:22:00 21118:21118 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 10:22:00 21117:21117 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.0048630Z 
2022-12-01T10:31:02.0048870Z STAGE:2022-12-01 10:22:00 21116:21116 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0049105Z STAGE:2022-12-01 10:22:00 21117:21117 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0049344Z STAGE:2022-12-01 10:22:00 21118:21118 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0049577Z STAGE:2022-12-01 10:22:00 21118:21118 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.0050021Z STAGE:2022-12-01 10:22:00 21117:21117 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 10:22:00 21116:21116 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.0050052Z 
2022-12-01T10:31:02.0050120Z ok (1.853s)
2022-12-01T10:31:02.0050125Z 
2022-12-01T10:31:02.0050322Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0050397Z Ran 1 test in 1.853s
2022-12-01T10:31:02.0050402Z 
2022-12-01T10:31:02.0050463Z OK
2022-12-01T10:31:02.0050468Z 
2022-12-01T10:31:02.0050553Z Generating XML reports...
2022-12-01T10:31:02.0050894Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102159.xml
2022-12-01T10:31:02.0051130Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.0051392Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0051522Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0051809Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0051945Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0051949Z 
2022-12-01T10:31:02.0052022Z Running tests...
2022-12-01T10:31:02.0052220Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0052453Z   test_all_reduce_multigpu (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21172
2022-12-01T10:31:02.0052610Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21173
2022-12-01T10:31:02.0052766Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 21174
2022-12-01T10:31:02.0053023Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0053149Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0053430Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0053569Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0053747Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:31:02.0054045Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0054314Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0054436Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0054702Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0054839Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0055017Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:31:02.0055364Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0055490Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0055790Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0055928Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0056107Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:31:02.0056405Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0056732Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0056929Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:31:02.0057093Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:31:02.0057258Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:31:02.0057354Z skip: CUDA is not available. (1.852s)
2022-12-01T10:31:02.0057359Z 
2022-12-01T10:31:02.0057554Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0057628Z Ran 1 test in 1.852s
2022-12-01T10:31:02.0057633Z 
2022-12-01T10:31:02.0057705Z OK (skipped=1)
2022-12-01T10:31:02.0057709Z 
2022-12-01T10:31:02.0057794Z Generating XML reports...
2022-12-01T10:31:02.0058122Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102202.xml
2022-12-01T10:31:02.0058358Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.0058634Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0058757Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0059041Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0059180Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0059185Z 
2022-12-01T10:31:02.0059257Z Running tests...
2022-12-01T10:31:02.0059452Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0059682Z   test_all_reduce_multigpu_complex (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21225
2022-12-01T10:31:02.0059841Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21226
2022-12-01T10:31:02.0059999Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 21227
2022-12-01T10:31:02.0060273Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0060541Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0060835Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0060970Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0061149Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:31:02.0061424Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0061533Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0061819Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0061957Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0062136Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:31:02.0062409Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0062535Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0062821Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0062958Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0063137Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:31:02.0063497Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0063833Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0064126Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0064290Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:31:02.0064451Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:31:02.0064618Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:31:02.0064713Z skip: CUDA is not available. (1.757s)
2022-12-01T10:31:02.0064719Z 
2022-12-01T10:31:02.0064917Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0064980Z Ran 1 test in 1.758s
2022-12-01T10:31:02.0065003Z 
2022-12-01T10:31:02.0065062Z OK (skipped=1)
2022-12-01T10:31:02.0065070Z 
2022-12-01T10:31:02.0065155Z Generating XML reports...
2022-12-01T10:31:02.0065495Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102205.xml
2022-12-01T10:31:02.0065728Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.0066005Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0066130Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0066410Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0066548Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0066554Z 
2022-12-01T10:31:02.0066613Z Running tests...
2022-12-01T10:31:02.0066812Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0067048Z   test_all_reduce_product (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21278
2022-12-01T10:31:02.0067207Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21279
2022-12-01T10:31:02.0067363Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 21280
2022-12-01T10:31:02.0067635Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0067758Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0068037Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0068170Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0068337Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:31:02.0068611Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0068735Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0069016Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0069150Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0069326Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:31:02.0069595Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0069718Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0070015Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0070150Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0070384Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:31:02.0070684Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0070981Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0071278Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0071448Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:31:02.0071698Z STAGE:2022-12-01 10:22:10 21279:21279 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0071868Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:31:02.0072014Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:31:02.0072262Z STAGE:2022-12-01 10:22:10 21278:21278 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0072499Z STAGE:2022-12-01 10:22:10 21280:21280 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0072748Z STAGE:2022-12-01 10:22:10 21278:21278 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.0072989Z STAGE:2022-12-01 10:22:10 21280:21280 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.0073235Z STAGE:2022-12-01 10:22:10 21279:21279 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.0073475Z STAGE:2022-12-01 10:22:10 21278:21278 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0073715Z STAGE:2022-12-01 10:22:10 21280:21280 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0073952Z STAGE:2022-12-01 10:22:10 21279:21279 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0074184Z STAGE:2022-12-01 10:22:10 21278:21278 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.0074595Z STAGE:2022-12-01 10:22:10 21280:21280 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 10:22:10 21279:21279 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.0074601Z 
2022-12-01T10:31:02.0074838Z STAGE:2022-12-01 10:22:10 21278:21278 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0075237Z STAGE:2022-12-01 10:22:10 21279:21279 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-12-01 10:22:10 21280:21280 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0075242Z 
2022-12-01T10:31:02.0075491Z STAGE:2022-12-01 10:22:10 21278:21278 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.0075733Z STAGE:2022-12-01 10:22:10 21279:21279 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.0075974Z STAGE:2022-12-01 10:22:10 21280:21280 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.0076041Z ok (1.955s)
2022-12-01T10:31:02.0076046Z 
2022-12-01T10:31:02.0076238Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0076313Z Ran 1 test in 1.955s
2022-12-01T10:31:02.0076318Z 
2022-12-01T10:31:02.0076366Z OK
2022-12-01T10:31:02.0076382Z 
2022-12-01T10:31:02.0076452Z Generating XML reports...
2022-12-01T10:31:02.0076794Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102208.xml
2022-12-01T10:31:02.0077027Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.0077336Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0077492Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0077776Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0077912Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0077917Z 
2022-12-01T10:31:02.0077989Z Running tests...
2022-12-01T10:31:02.0078171Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0078408Z   test_all_reduce_result_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21334
2022-12-01T10:31:02.0078564Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21335
2022-12-01T10:31:02.0078721Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 21336
2022-12-01T10:31:02.0078995Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0079120Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0079400Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0079535Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0079712Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:31:02.0079968Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0080091Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0080368Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0080504Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0080686Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:31:02.0080985Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0081252Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0081376Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0081642Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0081777Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0081952Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:31:02.0082251Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0082549Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0082711Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:31:02.0082875Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:31:02.0083035Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:31:02.0083130Z skip: CUDA is not available. (1.756s)
2022-12-01T10:31:02.0083134Z 
2022-12-01T10:31:02.0083316Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0083391Z Ran 1 test in 1.757s
2022-12-01T10:31:02.0083397Z 
2022-12-01T10:31:02.0083467Z OK (skipped=1)
2022-12-01T10:31:02.0083472Z 
2022-12-01T10:31:02.0083555Z Generating XML reports...
2022-12-01T10:31:02.0083925Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102212.xml
2022-12-01T10:31:02.0084185Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.0084459Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0084582Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0084862Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0084984Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0084989Z 
2022-12-01T10:31:02.0085062Z Running tests...
2022-12-01T10:31:02.0085255Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0085486Z   test_all_reduce_sum (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21387
2022-12-01T10:31:02.0085644Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21388
2022-12-01T10:31:02.0085801Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 21389
2022-12-01T10:31:02.0086073Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0086198Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0086464Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0086600Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0086776Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:31:02.0087076Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0087348Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0087473Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0087751Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0087885Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0088063Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:31:02.0088320Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0088444Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0088724Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0088858Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0089036Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:31:02.0089334Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0089632Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0089797Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:31:02.0089959Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:31:02.0090103Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:31:02.0090544Z STAGE:2022-12-01 10:22:16 21388:21388 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-12-01 10:22:16 21387:21387 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0090575Z 
2022-12-01T10:31:02.0090821Z STAGE:2022-12-01 10:22:16 21389:21389 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0091070Z STAGE:2022-12-01 10:22:16 21387:21387 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.0091320Z STAGE:2022-12-01 10:22:16 21389:21389 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.0091563Z STAGE:2022-12-01 10:22:16 21388:21388 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.0091806Z STAGE:2022-12-01 10:22:16 21389:21389 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0092045Z STAGE:2022-12-01 10:22:16 21387:21387 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0092282Z STAGE:2022-12-01 10:22:16 21388:21388 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0092529Z STAGE:2022-12-01 10:22:16 21387:21387 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.0092760Z STAGE:2022-12-01 10:22:16 21389:21389 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.0093001Z STAGE:2022-12-01 10:22:16 21388:21388 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.0093240Z STAGE:2022-12-01 10:22:16 21388:21388 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0093477Z STAGE:2022-12-01 10:22:16 21389:21389 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0093719Z STAGE:2022-12-01 10:22:16 21387:21387 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0093964Z STAGE:2022-12-01 10:22:16 21389:21389 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.0094209Z STAGE:2022-12-01 10:22:16 21388:21388 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.0094450Z STAGE:2022-12-01 10:22:16 21387:21387 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.0094505Z ok (1.853s)
2022-12-01T10:31:02.0094523Z 
2022-12-01T10:31:02.0094705Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0094780Z Ran 1 test in 1.853s
2022-12-01T10:31:02.0094821Z 
2022-12-01T10:31:02.0094883Z OK
2022-12-01T10:31:02.0094888Z 
2022-12-01T10:31:02.0094972Z Generating XML reports...
2022-12-01T10:31:02.0095469Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102215.xml
2022-12-01T10:31:02.0095710Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.0095985Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0096113Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0096381Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0096517Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0096522Z 
2022-12-01T10:31:02.0096596Z Running tests...
2022-12-01T10:31:02.0096794Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0097029Z   test_all_reduce_sum_async (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21443
2022-12-01T10:31:02.0097188Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21444
2022-12-01T10:31:02.0097344Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 21445
2022-12-01T10:31:02.0097616Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0097774Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0098086Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0098224Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0098402Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:31:02.0098673Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0098798Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0099076Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0099210Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0099390Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:31:02.0099649Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0099771Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0100049Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0100185Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0100495Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:31:02.0100820Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0101118Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0101412Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0101583Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:31:02.0101733Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:31:02.0101983Z STAGE:2022-12-01 10:22:20 21445:21445 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0102225Z STAGE:2022-12-01 10:22:20 21444:21444 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0102390Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:31:02.0102630Z STAGE:2022-12-01 10:22:20 21443:21443 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0102876Z STAGE:2022-12-01 10:22:20 21443:21443 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.0103122Z STAGE:2022-12-01 10:22:20 21445:21445 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.0103364Z STAGE:2022-12-01 10:22:20 21444:21444 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.0103600Z STAGE:2022-12-01 10:22:20 21443:21443 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0103832Z STAGE:2022-12-01 10:22:20 21445:21445 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0104069Z STAGE:2022-12-01 10:22:20 21444:21444 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0104480Z STAGE:2022-12-01 10:22:20 21445:21445 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 10:22:20 21443:21443 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.0104730Z STAGE:2022-12-01 10:22:20 21444:21444 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.0104735Z 
2022-12-01T10:31:02.0105340Z STAGE:2022-12-01 10:22:20 21443:21443 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-12-01 10:22:20 21444:21444 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-12-01 10:22:20 21445:21445 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0105377Z 
2022-12-01T10:31:02.0105382Z 
2022-12-01T10:31:02.0105793Z STAGE:2022-12-01 10:22:20 21445:21445 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 10:22:20 21443:21443 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.0106041Z STAGE:2022-12-01 10:22:20 21444:21444 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.0106046Z 
2022-12-01T10:31:02.0106115Z ok (1.951s)
2022-12-01T10:31:02.0106119Z 
2022-12-01T10:31:02.0106318Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0106394Z Ran 1 test in 1.952s
2022-12-01T10:31:02.0106399Z 
2022-12-01T10:31:02.0106461Z OK
2022-12-01T10:31:02.0106468Z 
2022-12-01T10:31:02.0106553Z Generating XML reports...
2022-12-01T10:31:02.0106885Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102218.xml
2022-12-01T10:31:02.0107119Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.0107397Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0107523Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0107803Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0107940Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0107945Z 
2022-12-01T10:31:02.0108021Z Running tests...
2022-12-01T10:31:02.0108216Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0108443Z   test_all_reduce_sum_complex (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21499
2022-12-01T10:31:02.0108603Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21500
2022-12-01T10:31:02.0108759Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 21501
2022-12-01T10:31:02.0109033Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0109158Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0109440Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0109576Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0109755Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:31:02.0110025Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0110138Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0110419Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0110553Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0110735Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:31:02.0111031Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0111299Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0111424Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0111737Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0111897Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0112060Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:31:02.0112360Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0112651Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0112815Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:31:02.0112978Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:31:02.0113140Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:31:02.0113390Z STAGE:2022-12-01 10:22:23 21499:21499 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0113637Z STAGE:2022-12-01 10:22:23 21501:21501 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0113861Z STAGE:2022-12-01 10:22:23 21500:21500 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0114270Z STAGE:2022-12-01 10:22:23 21499:21499 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 10:22:23 21501:21501 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.0114276Z 
2022-12-01T10:31:02.0114522Z STAGE:2022-12-01 10:22:23 21500:21500 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.0114761Z STAGE:2022-12-01 10:22:23 21501:21501 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0115159Z STAGE:2022-12-01 10:22:23 21499:21499 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-12-01 10:22:23 21500:21500 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0115166Z 
2022-12-01T10:31:02.0115411Z STAGE:2022-12-01 10:22:23 21499:21499 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.0115652Z STAGE:2022-12-01 10:22:23 21501:21501 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.0115894Z STAGE:2022-12-01 10:22:23 21501:21501 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0116129Z STAGE:2022-12-01 10:22:23 21499:21499 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0116373Z STAGE:2022-12-01 10:22:23 21500:21500 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.0116606Z STAGE:2022-12-01 10:22:23 21500:21500 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0116838Z STAGE:2022-12-01 10:22:23 21500:21500 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.0117083Z STAGE:2022-12-01 10:22:23 21499:21499 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.0117329Z STAGE:2022-12-01 10:22:23 21501:21501 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.0117399Z ok (1.860s)
2022-12-01T10:31:02.0117403Z 
2022-12-01T10:31:02.0117601Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0117676Z Ran 1 test in 1.860s
2022-12-01T10:31:02.0117681Z 
2022-12-01T10:31:02.0117741Z OK
2022-12-01T10:31:02.0117746Z 
2022-12-01T10:31:02.0117829Z Generating XML reports...
2022-12-01T10:31:02.0118158Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102222.xml
2022-12-01T10:31:02.0118389Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.0118690Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0118844Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0119126Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0119261Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0119266Z 
2022-12-01T10:31:02.0119339Z Running tests...
2022-12-01T10:31:02.0119536Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0119770Z   test_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21555
2022-12-01T10:31:02.0119914Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21556
2022-12-01T10:31:02.0120071Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 21557
2022-12-01T10:31:02.0120349Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0120473Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0120753Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0120887Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0121065Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:31:02.0121335Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0121458Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0121723Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0121860Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0122037Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:31:02.0122311Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0122434Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0122712Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0122846Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0123021Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:31:02.0123308Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0123610Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0123906Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0124075Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:31:02.0124241Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:31:02.0124403Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:31:02.0124498Z skip: CUDA is not available. (1.754s)
2022-12-01T10:31:02.0124503Z 
2022-12-01T10:31:02.0124699Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0124772Z Ran 1 test in 1.754s
2022-12-01T10:31:02.0124778Z 
2022-12-01T10:31:02.0124836Z OK (skipped=1)
2022-12-01T10:31:02.0124840Z 
2022-12-01T10:31:02.0124924Z Generating XML reports...
2022-12-01T10:31:02.0125296Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102225.xml
2022-12-01T10:31:02.0125559Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.0125834Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0125955Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0126232Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0126366Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0126372Z 
2022-12-01T10:31:02.0126444Z Running tests...
2022-12-01T10:31:02.0126627Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0126867Z   test_all_reduce_sum_cuda_async (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21608
2022-12-01T10:31:02.0127024Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21609
2022-12-01T10:31:02.0127181Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 21610
2022-12-01T10:31:02.0127454Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0127576Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0127855Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0127991Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0128153Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:31:02.0128423Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0128549Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0128829Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0128966Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0129143Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:31:02.0129413Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0129538Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0129820Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0129942Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0130120Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:31:02.0130421Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0130719Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0131008Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0131174Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:31:02.0131336Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:31:02.0131498Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:31:02.0131593Z skip: CUDA is not available. (1.955s)
2022-12-01T10:31:02.0131598Z 
2022-12-01T10:31:02.0131807Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0131885Z Ran 1 test in 1.955s
2022-12-01T10:31:02.0131916Z 
2022-12-01T10:31:02.0131988Z OK (skipped=1)
2022-12-01T10:31:02.0131993Z 
2022-12-01T10:31:02.0132078Z Generating XML reports...
2022-12-01T10:31:02.0132419Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102228.xml
2022-12-01T10:31:02.0132649Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.0132922Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0133045Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0133323Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0133445Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0133451Z 
2022-12-01T10:31:02.0133526Z Running tests...
2022-12-01T10:31:02.0133722Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0133962Z   test_all_reduce_sum_cuda_complex (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21661
2022-12-01T10:31:02.0134120Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21662
2022-12-01T10:31:02.0134275Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 21663
2022-12-01T10:31:02.0134547Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0134668Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0134936Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0135073Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0135324Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:31:02.0135603Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0135726Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0136005Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0136141Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0136318Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:31:02.0136587Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0136699Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0136983Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0137122Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0137296Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:31:02.0137594Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0137891Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0138184Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0138350Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:31:02.0138545Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:31:02.0138696Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:31:02.0138831Z skip: CUDA is not available. (1.854s)
2022-12-01T10:31:02.0138836Z 
2022-12-01T10:31:02.0139032Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0139108Z Ran 1 test in 1.854s
2022-12-01T10:31:02.0139113Z 
2022-12-01T10:31:02.0139183Z OK (skipped=1)
2022-12-01T10:31:02.0139188Z 
2022-12-01T10:31:02.0139272Z Generating XML reports...
2022-12-01T10:31:02.0139614Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102232.xml
2022-12-01T10:31:02.0139848Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.0140108Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0140235Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0140662Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0140800Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0140805Z 
2022-12-01T10:31:02.0140879Z Running tests...
2022-12-01T10:31:02.0141079Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0141259Z   test_all_to_all (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.001s)
2022-12-01T10:31:02.0141264Z 
2022-12-01T10:31:02.0141458Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0141537Z Ran 1 test in 0.002s
2022-12-01T10:31:02.0141541Z 
2022-12-01T10:31:02.0141600Z OK (skipped=1)
2022-12-01T10:31:02.0141604Z 
2022-12-01T10:31:02.0141689Z Generating XML reports...
2022-12-01T10:31:02.0142030Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102235.xml
2022-12-01T10:31:02.0142264Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.0142540Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0142665Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0142948Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0143086Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0143091Z 
2022-12-01T10:31:02.0143164Z Running tests...
2022-12-01T10:31:02.0143345Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0143533Z   test_all_to_all_complex (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.001s)
2022-12-01T10:31:02.0143540Z 
2022-12-01T10:31:02.0143736Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0143812Z Ran 1 test in 0.002s
2022-12-01T10:31:02.0143818Z 
2022-12-01T10:31:02.0143888Z OK (skipped=1)
2022-12-01T10:31:02.0143892Z 
2022-12-01T10:31:02.0143975Z Generating XML reports...
2022-12-01T10:31:02.0144312Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102236.xml
2022-12-01T10:31:02.0144544Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.0144801Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0144924Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0145203Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0145400Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0145438Z 
2022-12-01T10:31:02.0145514Z Running tests...
2022-12-01T10:31:02.0145709Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0145899Z   test_all_to_all_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only NCCL supports CUDA all_to_all (0.001s)
2022-12-01T10:31:02.0145904Z 
2022-12-01T10:31:02.0146095Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0146172Z Ran 1 test in 0.002s
2022-12-01T10:31:02.0146177Z 
2022-12-01T10:31:02.0146233Z OK (skipped=1)
2022-12-01T10:31:02.0146237Z 
2022-12-01T10:31:02.0146322Z Generating XML reports...
2022-12-01T10:31:02.0146658Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102238.xml
2022-12-01T10:31:02.0146890Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.0147167Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0147293Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0147575Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0147712Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0147717Z 
2022-12-01T10:31:02.0147790Z Running tests...
2022-12-01T10:31:02.0147970Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0148170Z   test_all_to_all_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip: Only NCCL supports CUDA all_to_all (0.002s)
2022-12-01T10:31:02.0148175Z 
2022-12-01T10:31:02.0148369Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0148444Z Ran 1 test in 0.002s
2022-12-01T10:31:02.0148448Z 
2022-12-01T10:31:02.0148522Z OK (skipped=1)
2022-12-01T10:31:02.0148527Z 
2022-12-01T10:31:02.0148613Z Generating XML reports...
2022-12-01T10:31:02.0148948Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102239.xml
2022-12-01T10:31:02.0149176Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.0149446Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0149557Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0149838Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0149972Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0149977Z 
2022-12-01T10:31:02.0150052Z Running tests...
2022-12-01T10:31:02.0150246Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0150436Z   test_all_to_all_full_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.001s)
2022-12-01T10:31:02.0150443Z 
2022-12-01T10:31:02.0150635Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0150708Z Ran 1 test in 0.002s
2022-12-01T10:31:02.0150713Z 
2022-12-01T10:31:02.0150770Z OK (skipped=1)
2022-12-01T10:31:02.0150788Z 
2022-12-01T10:31:02.0150857Z Generating XML reports...
2022-12-01T10:31:02.0151196Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102240.xml
2022-12-01T10:31:02.0151427Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.0151699Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0151824Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0152132Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0152293Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0152298Z 
2022-12-01T10:31:02.0152370Z Running tests...
2022-12-01T10:31:02.0152550Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0152748Z   test_all_to_all_full_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only NCCL supports CUDA all_to_all (0.002s)
2022-12-01T10:31:02.0152753Z 
2022-12-01T10:31:02.0152943Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0153016Z Ran 1 test in 0.002s
2022-12-01T10:31:02.0153021Z 
2022-12-01T10:31:02.0153091Z OK (skipped=1)
2022-12-01T10:31:02.0153095Z 
2022-12-01T10:31:02.0153177Z Generating XML reports...
2022-12-01T10:31:02.0153518Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102242.xml
2022-12-01T10:31:02.0153748Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.0154021Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0154132Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0154413Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0154550Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0154554Z 
2022-12-01T10:31:02.0154626Z Running tests...
2022-12-01T10:31:02.0154821Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0155002Z   test_all_to_all_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.001s)
2022-12-01T10:31:02.0155008Z 
2022-12-01T10:31:02.0155201Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0155276Z Ran 1 test in 0.002s
2022-12-01T10:31:02.0155281Z 
2022-12-01T10:31:02.0155337Z OK (skipped=1)
2022-12-01T10:31:02.0155355Z 
2022-12-01T10:31:02.0155424Z Generating XML reports...
2022-12-01T10:31:02.0155760Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102243.xml
2022-12-01T10:31:02.0155990Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.0156263Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0156387Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0156667Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0156802Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0156809Z 
2022-12-01T10:31:02.0156913Z Running tests...
2022-12-01T10:31:02.0157096Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0157297Z   test_all_to_all_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-12-01T10:31:02.0157302Z 
2022-12-01T10:31:02.0157492Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0157566Z Ran 1 test in 0.002s
2022-12-01T10:31:02.0157570Z 
2022-12-01T10:31:02.0157641Z OK (skipped=1)
2022-12-01T10:31:02.0157646Z 
2022-12-01T10:31:02.0157729Z Generating XML reports...
2022-12-01T10:31:02.0158067Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102244.xml
2022-12-01T10:31:02.0158299Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.0158606Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0158743Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0159023Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0159157Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0159165Z 
2022-12-01T10:31:02.0159238Z Running tests...
2022-12-01T10:31:02.0159430Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0159639Z   test_all_to_all_single_equal_split (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.001s)
2022-12-01T10:31:02.0159644Z 
2022-12-01T10:31:02.0159835Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0159910Z Ran 1 test in 0.002s
2022-12-01T10:31:02.0159915Z 
2022-12-01T10:31:02.0159984Z OK (skipped=1)
2022-12-01T10:31:02.0159991Z 
2022-12-01T10:31:02.0160061Z Generating XML reports...
2022-12-01T10:31:02.0160402Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102246.xml
2022-12-01T10:31:02.0160632Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.0160911Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0161038Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0161319Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0161455Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0161460Z 
2022-12-01T10:31:02.0161533Z Running tests...
2022-12-01T10:31:02.0161713Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0161935Z   test_all_to_all_single_equal_split_complex (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.001s)
2022-12-01T10:31:02.0161942Z 
2022-12-01T10:31:02.0162132Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0162207Z Ran 1 test in 0.002s
2022-12-01T10:31:02.0162212Z 
2022-12-01T10:31:02.0162281Z OK (skipped=1)
2022-12-01T10:31:02.0162286Z 
2022-12-01T10:31:02.0162369Z Generating XML reports...
2022-12-01T10:31:02.0162708Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102247.xml
2022-12-01T10:31:02.0162942Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.0163214Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0163324Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0163608Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0163744Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0163748Z 
2022-12-01T10:31:02.0163820Z Running tests...
2022-12-01T10:31:02.0164014Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0164226Z   test_all_to_all_single_equal_split_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-12-01T10:31:02.0164231Z 
2022-12-01T10:31:02.0164423Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0164498Z Ran 1 test in 0.002s
2022-12-01T10:31:02.0164503Z 
2022-12-01T10:31:02.0164572Z OK (skipped=1)
2022-12-01T10:31:02.0164577Z 
2022-12-01T10:31:02.0164647Z Generating XML reports...
2022-12-01T10:31:02.0165020Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102248.xml
2022-12-01T10:31:02.0165279Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.0165551Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0165674Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0165956Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0166092Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0166097Z 
2022-12-01T10:31:02.0166169Z Running tests...
2022-12-01T10:31:02.0166363Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0166570Z   test_all_to_all_single_equal_split_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-12-01T10:31:02.0166575Z 
2022-12-01T10:31:02.0166768Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0166845Z Ran 1 test in 0.002s
2022-12-01T10:31:02.0166850Z 
2022-12-01T10:31:02.0166920Z OK (skipped=1)
2022-12-01T10:31:02.0166925Z 
2022-12-01T10:31:02.0167007Z Generating XML reports...
2022-12-01T10:31:02.0167345Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102250.xml
2022-12-01T10:31:02.0167581Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.0167851Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0167972Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0168239Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0168377Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0168383Z 
2022-12-01T10:31:02.0168458Z Running tests...
2022-12-01T10:31:02.0168652Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0168867Z   test_all_to_all_single_equal_split_full_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.001s)
2022-12-01T10:31:02.0168872Z 
2022-12-01T10:31:02.0169069Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0169142Z Ran 1 test in 0.002s
2022-12-01T10:31:02.0169147Z 
2022-12-01T10:31:02.0169216Z OK (skipped=1)
2022-12-01T10:31:02.0169220Z 
2022-12-01T10:31:02.0169302Z Generating XML reports...
2022-12-01T10:31:02.0169628Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102251.xml
2022-12-01T10:31:02.0169860Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.0170136Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0170257Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0170542Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0170681Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0170686Z 
2022-12-01T10:31:02.0170760Z Running tests...
2022-12-01T10:31:02.0170954Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0171176Z   test_all_to_all_single_equal_split_full_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-12-01T10:31:02.0171181Z 
2022-12-01T10:31:02.0171374Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0171450Z Ran 1 test in 0.002s
2022-12-01T10:31:02.0171483Z 
2022-12-01T10:31:02.0171557Z OK (skipped=1)
2022-12-01T10:31:02.0171587Z 
2022-12-01T10:31:02.0171658Z Generating XML reports...
2022-12-01T10:31:02.0171999Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102252.xml
2022-12-01T10:31:02.0172231Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.0172503Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0172628Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0172906Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0173042Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0173046Z 
2022-12-01T10:31:02.0173119Z Running tests...
2022-12-01T10:31:02.0173315Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0173515Z   test_all_to_all_single_equal_split_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.001s)
2022-12-01T10:31:02.0173521Z 
2022-12-01T10:31:02.0173712Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0173787Z Ran 1 test in 0.002s
2022-12-01T10:31:02.0173792Z 
2022-12-01T10:31:02.0173864Z OK (skipped=1)
2022-12-01T10:31:02.0173869Z 
2022-12-01T10:31:02.0173953Z Generating XML reports...
2022-12-01T10:31:02.0174287Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102254.xml
2022-12-01T10:31:02.0174516Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.0174788Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0174914Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0175183Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0175413Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0175419Z 
2022-12-01T10:31:02.0175495Z Running tests...
2022-12-01T10:31:02.0175694Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0175915Z   test_all_to_all_single_equal_split_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-12-01T10:31:02.0175920Z 
2022-12-01T10:31:02.0176112Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0176187Z Ran 1 test in 0.002s
2022-12-01T10:31:02.0176191Z 
2022-12-01T10:31:02.0176262Z OK (skipped=1)
2022-12-01T10:31:02.0176266Z 
2022-12-01T10:31:02.0176336Z Generating XML reports...
2022-12-01T10:31:02.0176678Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102255.xml
2022-12-01T10:31:02.0176914Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.0177189Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0177313Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0177592Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0177728Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0177733Z 
2022-12-01T10:31:02.0177805Z Running tests...
2022-12-01T10:31:02.0178000Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0178254Z   test_all_to_all_single_unequal_split (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.001s)
2022-12-01T10:31:02.0178299Z 
2022-12-01T10:31:02.0178481Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0178554Z Ran 1 test in 0.002s
2022-12-01T10:31:02.0178559Z 
2022-12-01T10:31:02.0178629Z OK (skipped=1)
2022-12-01T10:31:02.0178633Z 
2022-12-01T10:31:02.0178718Z Generating XML reports...
2022-12-01T10:31:02.0179056Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102256.xml
2022-12-01T10:31:02.0179292Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.0179565Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0179688Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0179958Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0180095Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0180102Z 
2022-12-01T10:31:02.0180176Z Running tests...
2022-12-01T10:31:02.0180471Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0180692Z   test_all_to_all_single_unequal_split_complex (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.001s)
2022-12-01T10:31:02.0180698Z 
2022-12-01T10:31:02.0180891Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0180968Z Ran 1 test in 0.002s
2022-12-01T10:31:02.0180972Z 
2022-12-01T10:31:02.0181044Z OK (skipped=1)
2022-12-01T10:31:02.0181049Z 
2022-12-01T10:31:02.0181134Z Generating XML reports...
2022-12-01T10:31:02.0181456Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102258.xml
2022-12-01T10:31:02.0181689Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.0181964Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0182088Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0182370Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0182506Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0182511Z 
2022-12-01T10:31:02.0182584Z Running tests...
2022-12-01T10:31:02.0182775Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0182978Z   test_all_to_all_single_unequal_split_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-12-01T10:31:02.0182997Z 
2022-12-01T10:31:02.0183177Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0183252Z Ran 1 test in 0.002s
2022-12-01T10:31:02.0183259Z 
2022-12-01T10:31:02.0183331Z OK (skipped=1)
2022-12-01T10:31:02.0183335Z 
2022-12-01T10:31:02.0183419Z Generating XML reports...
2022-12-01T10:31:02.0183752Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102259.xml
2022-12-01T10:31:02.0183981Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.0184250Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0184373Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0184640Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0184774Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0184779Z 
2022-12-01T10:31:02.0184899Z Running tests...
2022-12-01T10:31:02.0185095Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0185355Z   test_all_to_all_single_unequal_split_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-12-01T10:31:02.0185363Z 
2022-12-01T10:31:02.0185558Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0185634Z Ran 1 test in 0.002s
2022-12-01T10:31:02.0185639Z 
2022-12-01T10:31:02.0185710Z OK (skipped=1)
2022-12-01T10:31:02.0185715Z 
2022-12-01T10:31:02.0185798Z Generating XML reports...
2022-12-01T10:31:02.0186121Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102300.xml
2022-12-01T10:31:02.0186357Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.0186634Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0186760Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0187042Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0187178Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0187183Z 
2022-12-01T10:31:02.0187256Z Running tests...
2022-12-01T10:31:02.0187450Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0187669Z   test_all_to_all_single_unequal_split_full_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.001s)
2022-12-01T10:31:02.0187675Z 
2022-12-01T10:31:02.0187849Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0187924Z Ran 1 test in 0.002s
2022-12-01T10:31:02.0187930Z 
2022-12-01T10:31:02.0188002Z OK (skipped=1)
2022-12-01T10:31:02.0188006Z 
2022-12-01T10:31:02.0188092Z Generating XML reports...
2022-12-01T10:31:02.0188430Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102302.xml
2022-12-01T10:31:02.0188662Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.0188934Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0189057Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0189334Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0189457Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0189462Z 
2022-12-01T10:31:02.0189533Z Running tests...
2022-12-01T10:31:02.0189728Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0189955Z   test_all_to_all_single_unequal_split_full_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-12-01T10:31:02.0189962Z 
2022-12-01T10:31:02.0190154Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0190228Z Ran 1 test in 0.002s
2022-12-01T10:31:02.0190233Z 
2022-12-01T10:31:02.0190303Z OK (skipped=1)
2022-12-01T10:31:02.0190308Z 
2022-12-01T10:31:02.0190389Z Generating XML reports...
2022-12-01T10:31:02.0190722Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102303.xml
2022-12-01T10:31:02.0190941Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.0191212Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0191336Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0191660Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0191836Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0191841Z 
2022-12-01T10:31:02.0191913Z Running tests...
2022-12-01T10:31:02.0192104Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0192317Z   test_all_to_all_single_unequal_split_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.001s)
2022-12-01T10:31:02.0192323Z 
2022-12-01T10:31:02.0192513Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0192576Z Ran 1 test in 0.002s
2022-12-01T10:31:02.0192581Z 
2022-12-01T10:31:02.0192651Z OK (skipped=1)
2022-12-01T10:31:02.0192656Z 
2022-12-01T10:31:02.0192738Z Generating XML reports...
2022-12-01T10:31:02.0193076Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102305.xml
2022-12-01T10:31:02.0193308Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.0193583Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0193708Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0193990Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0194111Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0194130Z 
2022-12-01T10:31:02.0194190Z Running tests...
2022-12-01T10:31:02.0194386Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0194610Z   test_all_to_all_single_unequal_split_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-12-01T10:31:02.0194618Z 
2022-12-01T10:31:02.0194811Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0194887Z Ran 1 test in 0.002s
2022-12-01T10:31:02.0194892Z 
2022-12-01T10:31:02.0194962Z OK (skipped=1)
2022-12-01T10:31:02.0194967Z 
2022-12-01T10:31:02.0195049Z Generating XML reports...
2022-12-01T10:31:02.0195381Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102306.xml
2022-12-01T10:31:02.0195596Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.0195869Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0195992Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0196271Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0196406Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0196413Z 
2022-12-01T10:31:02.0196484Z Running tests...
2022-12-01T10:31:02.0196678Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0196912Z   test_average_parameters (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21954
2022-12-01T10:31:02.0197066Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21955
2022-12-01T10:31:02.0197208Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 21956
2022-12-01T10:31:02.0197480Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0197602Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0197881Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0198044Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0198251Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:31:02.0198522Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0198646Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0198911Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0199043Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0199220Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:31:02.0199490Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0199617Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0199898Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0200032Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0200209Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:31:02.0200506Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0200790Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0201082Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0201246Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:31:02.0201412Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:31:02.0201577Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:31:02.0201681Z skip: Need at least 2 CUDA devices (1.753s)
2022-12-01T10:31:02.0201686Z 
2022-12-01T10:31:02.0201882Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0201956Z Ran 1 test in 1.753s
2022-12-01T10:31:02.0201961Z 
2022-12-01T10:31:02.0202032Z OK (skipped=1)
2022-12-01T10:31:02.0202037Z 
2022-12-01T10:31:02.0202107Z Generating XML reports...
2022-12-01T10:31:02.0202444Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102307.xml
2022-12-01T10:31:02.0202676Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.0202953Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0203076Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0203357Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0203492Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0203497Z 
2022-12-01T10:31:02.0203568Z Running tests...
2022-12-01T10:31:02.0203749Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0204055Z   test_backend_full_group (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'gloo', 'nccl'} (0.001s)
2022-12-01T10:31:02.0204060Z 
2022-12-01T10:31:02.0204251Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0204325Z Ran 1 test in 0.002s
2022-12-01T10:31:02.0204331Z 
2022-12-01T10:31:02.0204402Z OK (skipped=1)
2022-12-01T10:31:02.0204406Z 
2022-12-01T10:31:02.0204517Z Generating XML reports...
2022-12-01T10:31:02.0204857Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102310.xml
2022-12-01T10:31:02.0205115Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.0205384Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0205495Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0205770Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0205907Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0205912Z 
2022-12-01T10:31:02.0205983Z Running tests...
2022-12-01T10:31:02.0206174Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0206476Z   test_backend_group (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'gloo', 'nccl'} (0.001s)
2022-12-01T10:31:02.0206483Z 
2022-12-01T10:31:02.0206673Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0206747Z Ran 1 test in 0.002s
2022-12-01T10:31:02.0206752Z 
2022-12-01T10:31:02.0206821Z OK (skipped=1)
2022-12-01T10:31:02.0206826Z 
2022-12-01T10:31:02.0206896Z Generating XML reports...
2022-12-01T10:31:02.0207233Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102312.xml
2022-12-01T10:31:02.0207464Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.0207736Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0207859Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0208140Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0208276Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0208280Z 
2022-12-01T10:31:02.0208350Z Running tests...
2022-12-01T10:31:02.0208544Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0208754Z   test_barrier (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22027
2022-12-01T10:31:02.0208911Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22028
2022-12-01T10:31:02.0209063Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 22029
2022-12-01T10:31:02.0209335Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0209457Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0209737Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0209879Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0210058Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:31:02.0210317Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0210441Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0210716Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0210851Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0211028Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:31:02.0211327Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0211476Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0211753Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0211885Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0212049Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:31:02.0212347Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0212641Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0212932Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0213098Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:31:02.0213263Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:31:02.0213426Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:31:02.0213493Z ok (3.162s)
2022-12-01T10:31:02.0213499Z 
2022-12-01T10:31:02.0213692Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0213754Z Ran 1 test in 3.163s
2022-12-01T10:31:02.0213759Z 
2022-12-01T10:31:02.0213819Z OK
2022-12-01T10:31:02.0213824Z 
2022-12-01T10:31:02.0213909Z Generating XML reports...
2022-12-01T10:31:02.0214246Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102313.xml
2022-12-01T10:31:02.0214479Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.0214752Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0214877Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0215208Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0215410Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0215464Z 
2022-12-01T10:31:02.0215537Z Running tests...
2022-12-01T10:31:02.0215769Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0216010Z   test_barrier_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22080
2022-12-01T10:31:02.0216328Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22081
2022-12-01T10:31:02.0216514Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 22082
2022-12-01T10:31:02.0216826Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0216980Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0217293Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0217415Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0217625Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:31:02.0217949Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0218107Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0218448Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0218648Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0218882Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:31:02.0219183Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0219333Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0219601Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0219771Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0219977Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:31:02.0220319Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0220775Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0221099Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0221291Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:31:02.0221480Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:31:02.0221676Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:31:02.0221761Z skip: CUDA is not available. (1.751s)
2022-12-01T10:31:02.0221767Z 
2022-12-01T10:31:02.0222020Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0222122Z Ran 1 test in 1.751s
2022-12-01T10:31:02.0222128Z 
2022-12-01T10:31:02.0222227Z OK (skipped=1)
2022-12-01T10:31:02.0222232Z 
2022-12-01T10:31:02.0222364Z Generating XML reports...
2022-12-01T10:31:02.0222736Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102318.xml
2022-12-01T10:31:02.0222998Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.0223308Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0223419Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0223725Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0223887Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0223892Z 
2022-12-01T10:31:02.0223993Z Running tests...
2022-12-01T10:31:02.0224218Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0224502Z   test_barrier_full_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22133
2022-12-01T10:31:02.0224689Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22134
2022-12-01T10:31:02.0224882Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 22135
2022-12-01T10:31:02.0225183Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0225293Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0225616Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0225776Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0225981Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:31:02.0226336Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0226516Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0226864Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0227028Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0227234Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:31:02.0227492Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0227645Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0227950Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0228111Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0228326Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:31:02.0228678Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0229005Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0229328Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0229540Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:31:02.0229691Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:31:02.0229880Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:31:02.0230084Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T10:31:02.0230292Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T10:31:02.0230492Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-12-01T10:31:02.0230834Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:31:02.0231156Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:31:02.0231476Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:31:02.0231534Z ok (3.256s)
2022-12-01T10:31:02.0231593Z 
2022-12-01T10:31:02.0231873Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0231987Z Ran 1 test in 3.256s
2022-12-01T10:31:02.0231994Z 
2022-12-01T10:31:02.0232089Z OK
2022-12-01T10:31:02.0232098Z 
2022-12-01T10:31:02.0232209Z Generating XML reports...
2022-12-01T10:31:02.0232583Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102321.xml
2022-12-01T10:31:02.0232871Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.0233175Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0233347Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0233615Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0233778Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0233784Z 
2022-12-01T10:31:02.0233891Z Running tests...
2022-12-01T10:31:02.0234113Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0234425Z   test_barrier_full_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22195
2022-12-01T10:31:02.0234636Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22196
2022-12-01T10:31:02.0234839Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 22197
2022-12-01T10:31:02.0235143Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0235301Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0235568Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0235728Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0235931Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:31:02.0236236Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0236389Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0236701Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0236883Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0237120Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:31:02.0237408Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0237705Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0237858Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0238164Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0238326Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0238508Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:31:02.0252806Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0253189Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0253360Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:31:02.0253529Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:31:02.0253690Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:31:02.0253794Z skip: CUDA is not available. (1.752s)
2022-12-01T10:31:02.0253803Z 
2022-12-01T10:31:02.0254003Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0254082Z Ran 1 test in 1.752s
2022-12-01T10:31:02.0254087Z 
2022-12-01T10:31:02.0254147Z OK (skipped=1)
2022-12-01T10:31:02.0254152Z 
2022-12-01T10:31:02.0254238Z Generating XML reports...
2022-12-01T10:31:02.0254588Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102326.xml
2022-12-01T10:31:02.0254824Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.0255107Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0255373Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0255799Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0255939Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0255983Z 
2022-12-01T10:31:02.0256058Z Running tests...
2022-12-01T10:31:02.0256242Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0256471Z   test_barrier_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22248
2022-12-01T10:31:02.0256632Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22249
2022-12-01T10:31:02.0256789Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 22250
2022-12-01T10:31:02.0257063Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0257188Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0257475Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0257613Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0257779Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:31:02.0258052Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0258175Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0258457Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0258593Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0258773Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:31:02.0259047Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0259171Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0259453Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0259574Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0259750Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:31:02.0260048Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0260461Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0260759Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0260928Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:31:02.0261097Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:31:02.0261259Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:31:02.0261438Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T10:31:02.0261599Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-12-01T10:31:02.0261898Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:31:02.0262069Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T10:31:02.0262362Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:31:02.0262714Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:31:02.0262842Z ok (2.757s)
2022-12-01T10:31:02.0262848Z 
2022-12-01T10:31:02.0263048Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0263123Z Ran 1 test in 2.757s
2022-12-01T10:31:02.0263129Z 
2022-12-01T10:31:02.0263177Z OK
2022-12-01T10:31:02.0263195Z 
2022-12-01T10:31:02.0263266Z Generating XML reports...
2022-12-01T10:31:02.0263609Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102329.xml
2022-12-01T10:31:02.0263841Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.0264119Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0264279Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0264564Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0264704Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0264709Z 
2022-12-01T10:31:02.0264783Z Running tests...
2022-12-01T10:31:02.0264965Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0265202Z   test_barrier_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22307
2022-12-01T10:31:02.0265362Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22308
2022-12-01T10:31:02.0265519Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 22309
2022-12-01T10:31:02.0265796Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0265920Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0266204Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0266341Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0266520Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:31:02.0266777Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0266901Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0267180Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0267316Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0267492Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:31:02.0267766Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0267894Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0268172Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0268294Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0268470Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:31:02.0268770Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0269067Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0269386Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0269555Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:31:02.0269745Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:31:02.0269907Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:31:02.0270004Z skip: CUDA is not available. (1.857s)
2022-12-01T10:31:02.0270010Z 
2022-12-01T10:31:02.0270192Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0270267Z Ran 1 test in 1.857s
2022-12-01T10:31:02.0270272Z 
2022-12-01T10:31:02.0270343Z OK (skipped=1)
2022-12-01T10:31:02.0270348Z 
2022-12-01T10:31:02.0270433Z Generating XML reports...
2022-12-01T10:31:02.0270775Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102333.xml
2022-12-01T10:31:02.0271010Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.0271285Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0271412Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0271690Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0271812Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0271817Z 
2022-12-01T10:31:02.0271890Z Running tests...
2022-12-01T10:31:02.0272087Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0272327Z   test_barrier_timeout_full_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22360
2022-12-01T10:31:02.0272484Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22361
2022-12-01T10:31:02.0272641Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 22362
2022-12-01T10:31:02.0272914Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0273037Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0273303Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0273439Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0273617Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:31:02.0273887Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0274011Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0274292Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0274431Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0274609Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:31:02.0274879Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0274990Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0275267Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0275401Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0275577Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:31:02.0275903Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0276206Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0276526Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0276692Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:31:02.0276852Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:31:02.0277000Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:31:02.0277177Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T10:31:02.0277351Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-12-01T10:31:02.0277526Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T10:31:02.0277823Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:31:02.0278112Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:31:02.0278403Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:31:02.0278473Z ok (3.057s)
2022-12-01T10:31:02.0278478Z 
2022-12-01T10:31:02.0278673Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0278735Z Ran 1 test in 3.057s
2022-12-01T10:31:02.0278741Z 
2022-12-01T10:31:02.0278802Z OK
2022-12-01T10:31:02.0278806Z 
2022-12-01T10:31:02.0278889Z Generating XML reports...
2022-12-01T10:31:02.0279232Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102336.xml
2022-12-01T10:31:02.0279522Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.0279799Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0279924Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0280206Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0280328Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0280346Z 
2022-12-01T10:31:02.0280407Z Running tests...
2022-12-01T10:31:02.0280599Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0280911Z   test_barrier_timeout_global (__main__.TestDistBackendWithSpawn) ... skip: Requires file:// initialization method. Both tcp:// and env:// rely on the TCP store for which reinitialization has proven racy. (0.002s)
2022-12-01T10:31:02.0280917Z 
2022-12-01T10:31:02.0281110Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0281185Z Ran 1 test in 0.002s
2022-12-01T10:31:02.0281190Z 
2022-12-01T10:31:02.0281262Z OK (skipped=1)
2022-12-01T10:31:02.0281266Z 
2022-12-01T10:31:02.0281349Z Generating XML reports...
2022-12-01T10:31:02.0281688Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102341.xml
2022-12-01T10:31:02.0281920Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.0282180Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0282304Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0282588Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0282755Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0282785Z 
2022-12-01T10:31:02.0282860Z Running tests...
2022-12-01T10:31:02.0283055Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0283290Z   test_barrier_timeout_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22432
2022-12-01T10:31:02.0283492Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22433
2022-12-01T10:31:02.0283697Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 22434
2022-12-01T10:31:02.0284069Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0284195Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0284482Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0284618Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0284800Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:31:02.0285074Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0285198Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0285478Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0285601Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0285778Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:31:02.0286047Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0286174Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0286456Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0286591Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0286766Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:31:02.0287065Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0287363Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0287643Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0287811Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:31:02.0287974Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:31:02.0288139Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:31:02.0288312Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T10:31:02.0288486Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T10:31:02.0288657Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-12-01T10:31:02.0288950Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:31:02.0289244Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:31:02.0289568Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:31:02.0289666Z ok (6.961s)
2022-12-01T10:31:02.0289672Z 
2022-12-01T10:31:02.0289870Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0289945Z Ran 1 test in 6.961s
2022-12-01T10:31:02.0289951Z 
2022-12-01T10:31:02.0290012Z OK
2022-12-01T10:31:02.0290017Z 
2022-12-01T10:31:02.0290101Z Generating XML reports...
2022-12-01T10:31:02.0290442Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102342.xml
2022-12-01T10:31:02.0290676Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.0290937Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0291060Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0291342Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0291480Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0291485Z 
2022-12-01T10:31:02.0291559Z Running tests...
2022-12-01T10:31:02.0291755Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0291991Z   test_batch_isend_irecv_gloo (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22491
2022-12-01T10:31:02.0292149Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22492
2022-12-01T10:31:02.0292306Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 22493
2022-12-01T10:31:02.0292563Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0292689Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0292972Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0293109Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0293285Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:31:02.0293554Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0293677Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0293953Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0294124Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0294286Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:31:02.0294559Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0294685Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0294965Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0295100Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0295355Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:31:02.0295661Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0295961Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0296241Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0296441Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:31:02.0296639Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:31:02.0296800Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:31:02.0296869Z ok (1.957s)
2022-12-01T10:31:02.0296874Z 
2022-12-01T10:31:02.0297070Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0297147Z Ran 1 test in 1.957s
2022-12-01T10:31:02.0297152Z 
2022-12-01T10:31:02.0297212Z OK
2022-12-01T10:31:02.0297217Z 
2022-12-01T10:31:02.0297300Z Generating XML reports...
2022-12-01T10:31:02.0297632Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102351.xml
2022-12-01T10:31:02.0297867Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.0298143Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0298269Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0298552Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0298687Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0298692Z 
2022-12-01T10:31:02.0298765Z Running tests...
2022-12-01T10:31:02.0298960Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0299201Z   test_batch_isend_irecv_gloo_tags (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22544
2022-12-01T10:31:02.0299345Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22545
2022-12-01T10:31:02.0299501Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 22546
2022-12-01T10:31:02.0299776Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0299903Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0300185Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0300320Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0300710Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:31:02.0300983Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0301092Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0301377Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0301514Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0301694Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:31:02.0301969Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0302092Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0302373Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0302510Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0302686Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:31:02.0302971Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0303325Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0303650Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0303817Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:31:02.0303978Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:31:02.0304139Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:31:02.0304208Z ok (1.955s)
2022-12-01T10:31:02.0304213Z 
2022-12-01T10:31:02.0304411Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0304474Z Ran 1 test in 1.955s
2022-12-01T10:31:02.0304492Z 
2022-12-01T10:31:02.0304540Z OK
2022-12-01T10:31:02.0304544Z 
2022-12-01T10:31:02.0304630Z Generating XML reports...
2022-12-01T10:31:02.0304975Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102354.xml
2022-12-01T10:31:02.0305211Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.0305486Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0305610Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0305890Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0306050Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0306055Z 
2022-12-01T10:31:02.0306115Z Running tests...
2022-12-01T10:31:02.0306311Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0306516Z   test_batch_isend_irecv_mixed_backend_err (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s)
2022-12-01T10:31:02.0306523Z 
2022-12-01T10:31:02.0306715Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0306791Z Ran 1 test in 0.002s
2022-12-01T10:31:02.0306796Z 
2022-12-01T10:31:02.0306869Z OK (skipped=1)
2022-12-01T10:31:02.0306873Z 
2022-12-01T10:31:02.0306958Z Generating XML reports...
2022-12-01T10:31:02.0307299Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102358.xml
2022-12-01T10:31:02.0307529Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.0307788Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0307912Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0308198Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0308334Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0308340Z 
2022-12-01T10:31:02.0308414Z Running tests...
2022-12-01T10:31:02.0308610Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0308795Z   test_batch_isend_irecv_nccl (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s)
2022-12-01T10:31:02.0308800Z 
2022-12-01T10:31:02.0308990Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0309066Z Ran 1 test in 0.003s
2022-12-01T10:31:02.0309071Z 
2022-12-01T10:31:02.0309129Z OK (skipped=1)
2022-12-01T10:31:02.0309133Z 
2022-12-01T10:31:02.0309217Z Generating XML reports...
2022-12-01T10:31:02.0309552Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102359.xml
2022-12-01T10:31:02.0309810Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.0310087Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0310253Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0310533Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0310669Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0310674Z 
2022-12-01T10:31:02.0310733Z Running tests...
2022-12-01T10:31:02.0310927Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0311124Z   test_batch_isend_irecv_no_rank_zero_nccl (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s)
2022-12-01T10:31:02.0311128Z 
2022-12-01T10:31:02.0311321Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0311399Z Ran 1 test in 0.002s
2022-12-01T10:31:02.0311404Z 
2022-12-01T10:31:02.0311477Z OK (skipped=1)
2022-12-01T10:31:02.0311483Z 
2022-12-01T10:31:02.0311566Z Generating XML reports...
2022-12-01T10:31:02.0311904Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102400.xml
2022-12-01T10:31:02.0312136Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.0312396Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0312520Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0312800Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0312933Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0312938Z 
2022-12-01T10:31:02.0313011Z Running tests...
2022-12-01T10:31:02.0313207Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0313398Z   test_batch_isend_irecv_op_err (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s)
2022-12-01T10:31:02.0313402Z 
2022-12-01T10:31:02.0313593Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0313668Z Ran 1 test in 0.002s
2022-12-01T10:31:02.0313673Z 
2022-12-01T10:31:02.0313731Z OK (skipped=1)
2022-12-01T10:31:02.0313735Z 
2022-12-01T10:31:02.0313820Z Generating XML reports...
2022-12-01T10:31:02.0314160Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102402.xml
2022-12-01T10:31:02.0314394Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.0314666Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0314793Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0315076Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0315211Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0315216Z 
2022-12-01T10:31:02.0315288Z Running tests...
2022-12-01T10:31:02.0315469Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0315659Z   test_batch_isend_irecv_op_list_err (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s)
2022-12-01T10:31:02.0315664Z 
2022-12-01T10:31:02.0315857Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0315933Z Ran 1 test in 0.002s
2022-12-01T10:31:02.0315938Z 
2022-12-01T10:31:02.0316030Z OK (skipped=1)
2022-12-01T10:31:02.0316035Z 
2022-12-01T10:31:02.0316118Z Generating XML reports...
2022-12-01T10:31:02.0316482Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102403.xml
2022-12-01T10:31:02.0316738Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.0316998Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0317121Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0317402Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0317536Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0317541Z 
2022-12-01T10:31:02.0317614Z Running tests...
2022-12-01T10:31:02.0317806Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0318009Z   test_batch_isend_irecv_ring_exchange_nccl (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s)
2022-12-01T10:31:02.0318015Z 
2022-12-01T10:31:02.0318209Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0318284Z Ran 1 test in 0.002s
2022-12-01T10:31:02.0318288Z 
2022-12-01T10:31:02.0318346Z OK (skipped=1)
2022-12-01T10:31:02.0318350Z 
2022-12-01T10:31:02.0318433Z Generating XML reports...
2022-12-01T10:31:02.0318771Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102404.xml
2022-12-01T10:31:02.0319003Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.0319279Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0319402Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0319683Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0319822Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0319829Z 
2022-12-01T10:31:02.0319901Z Running tests...
2022-12-01T10:31:02.0320081Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0320273Z   test_batch_isend_irecv_self_nccl (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s)
2022-12-01T10:31:02.0320278Z 
2022-12-01T10:31:02.0320471Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0320546Z Ran 1 test in 0.002s
2022-12-01T10:31:02.0320551Z 
2022-12-01T10:31:02.0320622Z OK (skipped=1)
2022-12-01T10:31:02.0320627Z 
2022-12-01T10:31:02.0320710Z Generating XML reports...
2022-12-01T10:31:02.0321045Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102406.xml
2022-12-01T10:31:02.0321274Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.0321545Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0321657Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0321935Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0322070Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0322075Z 
2022-12-01T10:31:02.0322147Z Running tests...
2022-12-01T10:31:02.0322339Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0322531Z   test_batch_isend_irecv_tensor_err (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s)
2022-12-01T10:31:02.0322536Z 
2022-12-01T10:31:02.0322728Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0322803Z Ran 1 test in 0.002s
2022-12-01T10:31:02.0322808Z 
2022-12-01T10:31:02.0322916Z OK (skipped=1)
2022-12-01T10:31:02.0322966Z 
2022-12-01T10:31:02.0323037Z Generating XML reports...
2022-12-01T10:31:02.0323376Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102407.xml
2022-12-01T10:31:02.0323606Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.0323880Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0324003Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0324284Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0324420Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0324425Z 
2022-12-01T10:31:02.0324501Z Running tests...
2022-12-01T10:31:02.0324683Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0324911Z   test_broadcast (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22677
2022-12-01T10:31:02.0325068Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22678
2022-12-01T10:31:02.0325226Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 22679
2022-12-01T10:31:02.0325498Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0325621Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0325900Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0326035Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0326201Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:31:02.0326474Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0326600Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0326884Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0327019Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0327196Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:31:02.0327467Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0327592Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0327870Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0327992Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0328170Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:31:02.0328470Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0328767Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0329061Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0329226Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:31:02.0329390Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:31:02.0329672Z STAGE:2022-12-01 10:24:10 22678:22678 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0329923Z STAGE:2022-12-01 10:24:10 22679:22679 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0330101Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:31:02.0330342Z STAGE:2022-12-01 10:24:10 22677:22677 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0330591Z STAGE:2022-12-01 10:24:10 22677:22677 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.0330832Z STAGE:2022-12-01 10:24:10 22679:22679 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.0331079Z STAGE:2022-12-01 10:24:10 22678:22678 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.0331322Z STAGE:2022-12-01 10:24:10 22677:22677 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0331725Z STAGE:2022-12-01 10:24:10 22678:22678 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-12-01 10:24:10 22679:22679 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0331733Z 
2022-12-01T10:31:02.0332144Z STAGE:2022-12-01 10:24:10 22678:22678 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 10:24:10 22677:22677 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.0332150Z 
2022-12-01T10:31:02.0332394Z STAGE:2022-12-01 10:24:10 22679:22679 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.0332634Z STAGE:2022-12-01 10:24:10 22678:22678 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0332857Z STAGE:2022-12-01 10:24:10 22679:22679 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0333091Z STAGE:2022-12-01 10:24:10 22677:22677 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0333335Z STAGE:2022-12-01 10:24:10 22678:22678 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.0333585Z STAGE:2022-12-01 10:24:10 22679:22679 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.0333827Z STAGE:2022-12-01 10:24:10 22677:22677 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.0334069Z STAGE:2022-12-01 10:24:10 22678:22678 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0334306Z STAGE:2022-12-01 10:24:10 22679:22679 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0334541Z STAGE:2022-12-01 10:24:10 22677:22677 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0334789Z STAGE:2022-12-01 10:24:10 22677:22677 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.0335020Z STAGE:2022-12-01 10:24:10 22679:22679 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.0335356Z STAGE:2022-12-01 10:24:10 22678:22678 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.0335607Z STAGE:2022-12-01 10:24:10 22677:22677 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0335845Z STAGE:2022-12-01 10:24:10 22678:22678 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0336083Z STAGE:2022-12-01 10:24:10 22679:22679 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0336325Z STAGE:2022-12-01 10:24:10 22678:22678 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.0336566Z STAGE:2022-12-01 10:24:10 22677:22677 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.0336806Z STAGE:2022-12-01 10:24:10 22679:22679 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.0337043Z STAGE:2022-12-01 10:24:10 22678:22678 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0337465Z STAGE:2022-12-01 10:24:10 22679:22679 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-12-01 10:24:10 22677:22677 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0337511Z 
2022-12-01T10:31:02.0338065Z STAGE:2022-12-01 10:24:10 22677:22677 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 10:24:10 22679:22679 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 10:24:10 22678:22678 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.0338088Z 
2022-12-01T10:31:02.0338092Z 
2022-12-01T10:31:02.0338331Z STAGE:2022-12-01 10:24:10 22679:22679 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0338554Z STAGE:2022-12-01 10:24:10 22678:22678 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0338789Z STAGE:2022-12-01 10:24:10 22677:22677 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0339035Z STAGE:2022-12-01 10:24:10 22678:22678 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.0339279Z STAGE:2022-12-01 10:24:10 22679:22679 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.0339522Z STAGE:2022-12-01 10:24:10 22677:22677 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.0339761Z STAGE:2022-12-01 10:24:10 22677:22677 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0339999Z STAGE:2022-12-01 10:24:10 22678:22678 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0340233Z STAGE:2022-12-01 10:24:10 22679:22679 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0340759Z STAGE:2022-12-01 10:24:10 22677:22677 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 10:24:10 22679:22679 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.0340766Z 
2022-12-01T10:31:02.0340998Z STAGE:2022-12-01 10:24:10 22678:22678 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.0341239Z STAGE:2022-12-01 10:24:10 22678:22678 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0341477Z STAGE:2022-12-01 10:24:10 22679:22679 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0341712Z STAGE:2022-12-01 10:24:10 22677:22677 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0341956Z STAGE:2022-12-01 10:24:10 22678:22678 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.0342203Z STAGE:2022-12-01 10:24:10 22679:22679 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.0342444Z STAGE:2022-12-01 10:24:10 22677:22677 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.0342684Z STAGE:2022-12-01 10:24:10 22679:22679 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0342923Z STAGE:2022-12-01 10:24:10 22678:22678 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0343142Z STAGE:2022-12-01 10:24:10 22677:22677 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0343390Z STAGE:2022-12-01 10:24:10 22679:22679 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.0343634Z STAGE:2022-12-01 10:24:10 22677:22677 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.0343875Z STAGE:2022-12-01 10:24:10 22678:22678 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.0344115Z STAGE:2022-12-01 10:24:10 22677:22677 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0344351Z STAGE:2022-12-01 10:24:10 22679:22679 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0344591Z STAGE:2022-12-01 10:24:10 22678:22678 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0344900Z STAGE:2022-12-01 10:24:10 22678:22678 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.0345176Z STAGE:2022-12-01 10:24:10 22679:22679 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.0345402Z STAGE:2022-12-01 10:24:10 22677:22677 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.0345640Z STAGE:2022-12-01 10:24:10 22678:22678 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0345878Z STAGE:2022-12-01 10:24:10 22679:22679 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0346113Z STAGE:2022-12-01 10:24:10 22677:22677 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0346361Z STAGE:2022-12-01 10:24:10 22678:22678 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.0346602Z STAGE:2022-12-01 10:24:10 22679:22679 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.0346844Z STAGE:2022-12-01 10:24:10 22677:22677 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.0347085Z STAGE:2022-12-01 10:24:10 22679:22679 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0347321Z STAGE:2022-12-01 10:24:10 22678:22678 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0347543Z STAGE:2022-12-01 10:24:10 22677:22677 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0347785Z STAGE:2022-12-01 10:24:10 22679:22679 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.0348026Z STAGE:2022-12-01 10:24:10 22677:22677 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.0348268Z STAGE:2022-12-01 10:24:10 22678:22678 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.0348507Z STAGE:2022-12-01 10:24:10 22677:22677 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0348742Z STAGE:2022-12-01 10:24:10 22679:22679 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0348977Z STAGE:2022-12-01 10:24:10 22678:22678 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0349220Z STAGE:2022-12-01 10:24:10 22677:22677 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.0349464Z STAGE:2022-12-01 10:24:10 22678:22678 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.0349693Z STAGE:2022-12-01 10:24:10 22679:22679 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.0349932Z STAGE:2022-12-01 10:24:10 22678:22678 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0350170Z STAGE:2022-12-01 10:24:10 22677:22677 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0350408Z STAGE:2022-12-01 10:24:10 22679:22679 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0350649Z STAGE:2022-12-01 10:24:10 22677:22677 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.0350892Z STAGE:2022-12-01 10:24:10 22679:22679 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.0351130Z STAGE:2022-12-01 10:24:10 22678:22678 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.0351369Z STAGE:2022-12-01 10:24:10 22679:22679 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0351605Z STAGE:2022-12-01 10:24:10 22677:22677 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0351829Z STAGE:2022-12-01 10:24:10 22678:22678 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0352072Z STAGE:2022-12-01 10:24:10 22679:22679 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.0352340Z STAGE:2022-12-01 10:24:10 22677:22677 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.0352584Z STAGE:2022-12-01 10:24:10 22678:22678 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.0352848Z STAGE:2022-12-01 10:24:10 22677:22677 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0353084Z STAGE:2022-12-01 10:24:10 22679:22679 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0353319Z STAGE:2022-12-01 10:24:10 22678:22678 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0353563Z STAGE:2022-12-01 10:24:10 22677:22677 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.0353806Z STAGE:2022-12-01 10:24:10 22678:22678 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.0354033Z STAGE:2022-12-01 10:24:10 22679:22679 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.0354272Z STAGE:2022-12-01 10:24:10 22678:22678 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0354513Z STAGE:2022-12-01 10:24:10 22677:22677 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0354748Z STAGE:2022-12-01 10:24:10 22679:22679 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0355155Z STAGE:2022-12-01 10:24:10 22677:22677 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 10:24:10 22679:22679 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.0355161Z 
2022-12-01T10:31:02.0355402Z STAGE:2022-12-01 10:24:10 22678:22678 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.0355472Z ok (1.952s)
2022-12-01T10:31:02.0355477Z 
2022-12-01T10:31:02.0355674Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0355749Z Ran 1 test in 1.953s
2022-12-01T10:31:02.0355754Z 
2022-12-01T10:31:02.0355803Z OK
2022-12-01T10:31:02.0355807Z 
2022-12-01T10:31:02.0355895Z Generating XML reports...
2022-12-01T10:31:02.0356237Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102408.xml
2022-12-01T10:31:02.0356474Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.0356749Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0356874Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0357159Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0357294Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0357299Z 
2022-12-01T10:31:02.0357374Z Running tests...
2022-12-01T10:31:02.0357556Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0358243Z   test_broadcast_cuda (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/81028 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.236s)
2022-12-01T10:31:02.0358251Z 
2022-12-01T10:31:02.0358446Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0358522Z Ran 1 test in 0.236s
2022-12-01T10:31:02.0358527Z 
2022-12-01T10:31:02.0358604Z OK (skipped=1)
2022-12-01T10:31:02.0358609Z 
2022-12-01T10:31:02.0358693Z Generating XML reports...
2022-12-01T10:31:02.0359033Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102412.xml
2022-12-01T10:31:02.0359267Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.0359570Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0359718Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0359999Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0360135Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0360140Z 
2022-12-01T10:31:02.0360213Z Running tests...
2022-12-01T10:31:02.0360408Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0360645Z   test_broadcast_full_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22743
2022-12-01T10:31:02.0360802Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22744
2022-12-01T10:31:02.0360958Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 22745
2022-12-01T10:31:02.0361237Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0361348Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0361628Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0361763Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0361942Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:31:02.0362214Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0362338Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0362617Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0362756Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0362935Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:31:02.0363191Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0363312Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0363591Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0363728Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0363904Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:31:02.0364205Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0364505Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0364797Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0364967Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:31:02.0365118Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:31:02.0365279Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:31:02.0365454Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T10:31:02.0365627Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T10:31:02.0365799Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-12-01T10:31:02.0366127Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:31:02.0366449Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:31:02.0366736Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:31:02.0366987Z STAGE:2022-12-01 10:24:15 22743:22743 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0367216Z STAGE:2022-12-01 10:24:15 22745:22745 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0367466Z STAGE:2022-12-01 10:24:15 22745:22745 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.0367705Z STAGE:2022-12-01 10:24:15 22745:22745 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0367943Z STAGE:2022-12-01 10:24:15 22744:22744 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0368189Z STAGE:2022-12-01 10:24:15 22744:22744 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.0368434Z STAGE:2022-12-01 10:24:15 22743:22743 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.0368679Z STAGE:2022-12-01 10:24:15 22743:22743 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0368915Z STAGE:2022-12-01 10:24:15 22744:22744 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0369156Z STAGE:2022-12-01 10:24:15 22745:22745 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.0369388Z STAGE:2022-12-01 10:24:15 22744:22744 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.0369631Z STAGE:2022-12-01 10:24:15 22743:22743 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.0369876Z STAGE:2022-12-01 10:24:15 22744:22744 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0370115Z STAGE:2022-12-01 10:24:15 22745:22745 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0370361Z STAGE:2022-12-01 10:24:15 22744:22744 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.0370601Z STAGE:2022-12-01 10:24:15 22743:22743 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0370846Z STAGE:2022-12-01 10:24:15 22745:22745 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.0371088Z STAGE:2022-12-01 10:24:15 22743:22743 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.0371314Z STAGE:2022-12-01 10:24:15 22744:22744 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0371549Z STAGE:2022-12-01 10:24:15 22745:22745 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0371789Z STAGE:2022-12-01 10:24:15 22743:22743 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0372033Z STAGE:2022-12-01 10:24:15 22745:22745 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.0372274Z STAGE:2022-12-01 10:24:15 22743:22743 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.0372511Z STAGE:2022-12-01 10:24:15 22744:22744 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.0372748Z STAGE:2022-12-01 10:24:15 22743:22743 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0372986Z STAGE:2022-12-01 10:24:15 22745:22745 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0373221Z STAGE:2022-12-01 10:24:15 22744:22744 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0373452Z STAGE:2022-12-01 10:24:15 22745:22745 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.0373722Z STAGE:2022-12-01 10:24:15 22744:22744 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.0373989Z STAGE:2022-12-01 10:24:15 22743:22743 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.0374230Z STAGE:2022-12-01 10:24:15 22744:22744 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0374469Z STAGE:2022-12-01 10:24:15 22745:22745 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0374705Z STAGE:2022-12-01 10:24:15 22743:22743 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0374951Z STAGE:2022-12-01 10:24:15 22744:22744 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.0375192Z STAGE:2022-12-01 10:24:15 22745:22745 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.0375520Z STAGE:2022-12-01 10:24:15 22743:22743 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.0375748Z STAGE:2022-12-01 10:24:15 22745:22745 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0375987Z STAGE:2022-12-01 10:24:15 22744:22744 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0376221Z STAGE:2022-12-01 10:24:15 22743:22743 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0376466Z STAGE:2022-12-01 10:24:15 22743:22743 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.0376709Z STAGE:2022-12-01 10:24:15 22744:22744 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.0376950Z STAGE:2022-12-01 10:24:15 22745:22745 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.0377190Z STAGE:2022-12-01 10:24:15 22743:22743 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0377427Z STAGE:2022-12-01 10:24:15 22744:22744 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0377663Z STAGE:2022-12-01 10:24:15 22745:22745 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0377893Z STAGE:2022-12-01 10:24:15 22743:22743 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.0378134Z STAGE:2022-12-01 10:24:15 22744:22744 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.0378373Z STAGE:2022-12-01 10:24:15 22745:22745 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.0378615Z STAGE:2022-12-01 10:24:15 22744:22744 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0378850Z STAGE:2022-12-01 10:24:15 22743:22743 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0379087Z STAGE:2022-12-01 10:24:15 22745:22745 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0379332Z STAGE:2022-12-01 10:24:15 22745:22745 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.0379746Z STAGE:2022-12-01 10:24:15 22744:22744 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 10:24:15 22743:22743 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.0379754Z 
2022-12-01T10:31:02.0379992Z STAGE:2022-12-01 10:24:15 22745:22745 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0380229Z STAGE:2022-12-01 10:24:15 22744:22744 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0380562Z STAGE:2022-12-01 10:24:15 22743:22743 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0380807Z STAGE:2022-12-01 10:24:15 22743:22743 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.0381048Z STAGE:2022-12-01 10:24:15 22744:22744 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.0381287Z STAGE:2022-12-01 10:24:15 22745:22745 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.0381574Z STAGE:2022-12-01 10:24:15 22743:22743 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0381847Z STAGE:2022-12-01 10:24:15 22744:22744 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0382081Z STAGE:2022-12-01 10:24:15 22745:22745 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0382326Z STAGE:2022-12-01 10:24:15 22743:22743 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.0382567Z STAGE:2022-12-01 10:24:15 22744:22744 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.0382798Z STAGE:2022-12-01 10:24:15 22745:22745 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.0383036Z STAGE:2022-12-01 10:24:15 22744:22744 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0383275Z STAGE:2022-12-01 10:24:15 22743:22743 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0383513Z STAGE:2022-12-01 10:24:15 22745:22745 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0383760Z STAGE:2022-12-01 10:24:15 22743:22743 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.0384005Z STAGE:2022-12-01 10:24:15 22744:22744 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.0384245Z STAGE:2022-12-01 10:24:15 22745:22745 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.0384484Z STAGE:2022-12-01 10:24:15 22743:22743 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0384719Z STAGE:2022-12-01 10:24:15 22745:22745 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0384939Z STAGE:2022-12-01 10:24:15 22744:22744 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0385184Z STAGE:2022-12-01 10:24:15 22745:22745 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.0385424Z STAGE:2022-12-01 10:24:15 22743:22743 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.0385665Z STAGE:2022-12-01 10:24:15 22744:22744 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.0385904Z STAGE:2022-12-01 10:24:15 22743:22743 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0386139Z STAGE:2022-12-01 10:24:15 22745:22745 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0386378Z STAGE:2022-12-01 10:24:15 22744:22744 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0386622Z STAGE:2022-12-01 10:24:15 22743:22743 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.0386849Z STAGE:2022-12-01 10:24:15 22744:22744 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.0387091Z STAGE:2022-12-01 10:24:15 22745:22745 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.0387327Z STAGE:2022-12-01 10:24:15 22744:22744 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0387569Z STAGE:2022-12-01 10:24:15 22743:22743 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0387805Z STAGE:2022-12-01 10:24:15 22745:22745 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0388047Z STAGE:2022-12-01 10:24:15 22743:22743 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.0388287Z STAGE:2022-12-01 10:24:15 22745:22745 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.0388527Z STAGE:2022-12-01 10:24:15 22744:22744 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.0388956Z STAGE:2022-12-01 10:24:15 22745:22745 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-12-01 10:24:15 22743:22743 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0388963Z 
2022-12-01T10:31:02.0389231Z STAGE:2022-12-01 10:24:15 22744:22744 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0389459Z STAGE:2022-12-01 10:24:15 22745:22745 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.0389699Z STAGE:2022-12-01 10:24:15 22743:22743 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.0389941Z STAGE:2022-12-01 10:24:15 22744:22744 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.0390180Z STAGE:2022-12-01 10:24:15 22743:22743 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0390416Z STAGE:2022-12-01 10:24:15 22745:22745 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0390649Z STAGE:2022-12-01 10:24:15 22744:22744 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0390895Z STAGE:2022-12-01 10:24:15 22743:22743 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.0391141Z STAGE:2022-12-01 10:24:15 22744:22744 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.0391378Z STAGE:2022-12-01 10:24:15 22745:22745 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.0391605Z STAGE:2022-12-01 10:24:15 22744:22744 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0391838Z STAGE:2022-12-01 10:24:15 22743:22743 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0392073Z STAGE:2022-12-01 10:24:15 22745:22745 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0392316Z STAGE:2022-12-01 10:24:15 22743:22743 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.0392560Z STAGE:2022-12-01 10:24:15 22745:22745 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.0392804Z STAGE:2022-12-01 10:24:15 22744:22744 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.0392875Z ok (1.853s)
2022-12-01T10:31:02.0392880Z 
2022-12-01T10:31:02.0393077Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0393152Z Ran 1 test in 1.853s
2022-12-01T10:31:02.0393158Z 
2022-12-01T10:31:02.0393206Z OK
2022-12-01T10:31:02.0393210Z 
2022-12-01T10:31:02.0393295Z Generating XML reports...
2022-12-01T10:31:02.0393638Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102413.xml
2022-12-01T10:31:02.0393875Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.0394150Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0394276Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0394559Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0394698Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0394704Z 
2022-12-01T10:31:02.0394763Z Running tests...
2022-12-01T10:31:02.0394968Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0395201Z   test_broadcast_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22808
2022-12-01T10:31:02.0395358Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22809
2022-12-01T10:31:02.0395513Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 22810
2022-12-01T10:31:02.0395786Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0395912Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0396224Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0396387Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0396553Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:31:02.0396826Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0396949Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0397227Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0397364Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0397543Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:31:02.0397815Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0397940Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0398203Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0398337Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0398514Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:31:02.0398816Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0399115Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0399406Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0399575Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:31:02.0399738Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:31:02.0399914Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-12-01T10:31:02.0400074Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T10:31:02.0400237Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:31:02.0400405Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T10:31:02.0400696Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:31:02.0400990Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:31:02.0401281Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:31:02.0401535Z STAGE:2022-12-01 10:24:18 22809:22809 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0401778Z STAGE:2022-12-01 10:24:18 22810:22810 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0402026Z STAGE:2022-12-01 10:24:18 22810:22810 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.0402269Z STAGE:2022-12-01 10:24:18 22809:22809 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.0402494Z STAGE:2022-12-01 10:24:18 22809:22809 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0402729Z STAGE:2022-12-01 10:24:18 22810:22810 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0403004Z STAGE:2022-12-01 10:24:18 22810:22810 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.0403250Z STAGE:2022-12-01 10:24:18 22809:22809 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.0403521Z STAGE:2022-12-01 10:24:18 22810:22810 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0403759Z STAGE:2022-12-01 10:24:18 22809:22809 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0404003Z STAGE:2022-12-01 10:24:18 22809:22809 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.0404247Z STAGE:2022-12-01 10:24:18 22810:22810 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.0404474Z STAGE:2022-12-01 10:24:18 22809:22809 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0404710Z STAGE:2022-12-01 10:24:18 22810:22810 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0404958Z STAGE:2022-12-01 10:24:18 22810:22810 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.0405200Z STAGE:2022-12-01 10:24:18 22809:22809 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.0405441Z STAGE:2022-12-01 10:24:18 22810:22810 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0405678Z STAGE:2022-12-01 10:24:18 22809:22809 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0405924Z STAGE:2022-12-01 10:24:18 22809:22809 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.0406167Z STAGE:2022-12-01 10:24:18 22810:22810 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.0406406Z STAGE:2022-12-01 10:24:18 22809:22809 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0406630Z STAGE:2022-12-01 10:24:18 22810:22810 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0406877Z STAGE:2022-12-01 10:24:18 22810:22810 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.0407119Z STAGE:2022-12-01 10:24:18 22809:22809 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.0407358Z STAGE:2022-12-01 10:24:18 22810:22810 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0407593Z STAGE:2022-12-01 10:24:18 22809:22809 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0407837Z STAGE:2022-12-01 10:24:18 22809:22809 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.0408075Z STAGE:2022-12-01 10:24:18 22810:22810 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.0408314Z STAGE:2022-12-01 10:24:18 22809:22809 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0408548Z STAGE:2022-12-01 10:24:18 22810:22810 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0408777Z STAGE:2022-12-01 10:24:18 22810:22810 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.0409023Z STAGE:2022-12-01 10:24:18 22809:22809 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.0409261Z STAGE:2022-12-01 10:24:18 22810:22810 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0409496Z STAGE:2022-12-01 10:24:18 22809:22809 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0409739Z STAGE:2022-12-01 10:24:18 22809:22809 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.0409979Z STAGE:2022-12-01 10:24:18 22810:22810 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.0410219Z STAGE:2022-12-01 10:24:18 22809:22809 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0410456Z STAGE:2022-12-01 10:24:18 22810:22810 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0410728Z STAGE:2022-12-01 10:24:18 22810:22810 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.0410991Z STAGE:2022-12-01 10:24:18 22809:22809 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.0411229Z STAGE:2022-12-01 10:24:18 22810:22810 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0411466Z STAGE:2022-12-01 10:24:18 22809:22809 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0411711Z STAGE:2022-12-01 10:24:18 22809:22809 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.0411952Z STAGE:2022-12-01 10:24:18 22810:22810 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.0412189Z STAGE:2022-12-01 10:24:18 22809:22809 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0412423Z STAGE:2022-12-01 10:24:18 22810:22810 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0412667Z STAGE:2022-12-01 10:24:18 22810:22810 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.0412909Z STAGE:2022-12-01 10:24:18 22809:22809 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.0412965Z ok (1.952s)
2022-12-01T10:31:02.0412970Z 
2022-12-01T10:31:02.0413167Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0413242Z Ran 1 test in 1.952s
2022-12-01T10:31:02.0413247Z 
2022-12-01T10:31:02.0413309Z OK
2022-12-01T10:31:02.0413314Z 
2022-12-01T10:31:02.0413399Z Generating XML reports...
2022-12-01T10:31:02.0413744Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102417.xml
2022-12-01T10:31:02.0413978Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.0414255Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0414367Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0414653Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0414788Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0414792Z 
2022-12-01T10:31:02.0414865Z Running tests...
2022-12-01T10:31:02.0415060Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0415378Z   test_broadcast_multigpu (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22869
2022-12-01T10:31:02.0415540Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22870
2022-12-01T10:31:02.0415695Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 22871
2022-12-01T10:31:02.0415974Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0416086Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0416367Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0416503Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0416682Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:31:02.0416955Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0417080Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0417363Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0417498Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0417721Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:31:02.0418010Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0418134Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0418413Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0418548Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0418725Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:31:02.0419026Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0419318Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0419617Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0419784Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:31:02.0419938Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:31:02.0420097Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:31:02.0420195Z skip: CUDA is not available. (1.755s)
2022-12-01T10:31:02.0420200Z 
2022-12-01T10:31:02.0420499Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0420579Z Ran 1 test in 1.755s
2022-12-01T10:31:02.0420584Z 
2022-12-01T10:31:02.0420656Z OK (skipped=1)
2022-12-01T10:31:02.0420660Z 
2022-12-01T10:31:02.0420747Z Generating XML reports...
2022-12-01T10:31:02.0421095Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102420.xml
2022-12-01T10:31:02.0421315Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.0421592Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0421716Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0422000Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0422136Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0422140Z 
2022-12-01T10:31:02.0422216Z Running tests...
2022-12-01T10:31:02.0422413Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0423107Z   test_broadcast_object_list (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/82847 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.237s)
2022-12-01T10:31:02.0423114Z 
2022-12-01T10:31:02.0423309Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0423385Z Ran 1 test in 0.238s
2022-12-01T10:31:02.0423390Z 
2022-12-01T10:31:02.0423447Z OK (skipped=1)
2022-12-01T10:31:02.0423452Z 
2022-12-01T10:31:02.0423539Z Generating XML reports...
2022-12-01T10:31:02.0423877Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102423.xml
2022-12-01T10:31:02.0424109Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.0424382Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0424553Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0424841Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0425013Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0425018Z 
2022-12-01T10:31:02.0425091Z Running tests...
2022-12-01T10:31:02.0425274Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0425643Z   test_compute_bucket_assignment_by_size_sparse_error_with_logger (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'nccl', 'gloo'} (0.001s)
2022-12-01T10:31:02.0425649Z 
2022-12-01T10:31:02.0425843Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0425920Z Ran 1 test in 0.002s
2022-12-01T10:31:02.0425925Z 
2022-12-01T10:31:02.0425997Z OK (skipped=1)
2022-12-01T10:31:02.0426001Z 
2022-12-01T10:31:02.0426086Z Generating XML reports...
2022-12-01T10:31:02.0426427Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102425.xml
2022-12-01T10:31:02.0426663Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.0426939Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0427049Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0427332Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0427467Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0427472Z 
2022-12-01T10:31:02.0427544Z Running tests...
2022-12-01T10:31:02.0427737Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0428112Z   test_compute_bucket_assignment_by_size_sparse_error_without_logger (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'gloo', 'nccl'} (0.001s)
2022-12-01T10:31:02.0428119Z 
2022-12-01T10:31:02.0428309Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0428383Z Ran 1 test in 0.002s
2022-12-01T10:31:02.0428388Z 
2022-12-01T10:31:02.0428459Z OK (skipped=1)
2022-12-01T10:31:02.0428463Z 
2022-12-01T10:31:02.0428533Z Generating XML reports...
2022-12-01T10:31:02.0428872Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102426.xml
2022-12-01T10:31:02.0429102Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.0429377Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0429501Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0429783Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0429920Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0429925Z 
2022-12-01T10:31:02.0429997Z Running tests...
2022-12-01T10:31:02.0430180Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0430417Z   test_ddp_broadcast_buffer (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22952
2022-12-01T10:31:02.0430574Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22953
2022-12-01T10:31:02.0430728Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 22954
2022-12-01T10:31:02.0431004Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0431129Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0431440Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0431623Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0431801Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:31:02.0432059Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0432182Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0432461Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0432594Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0432771Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:31:02.0433043Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0433167Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0433447Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0433568Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0433744Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:31:02.0434045Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0434344Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0434633Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0434802Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:31:02.0434969Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:31:02.0435134Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:31:02.0435241Z skip: Need at least 2 CUDA devices (1.858s)
2022-12-01T10:31:02.0435246Z 
2022-12-01T10:31:02.0435442Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0435504Z Ran 1 test in 1.859s
2022-12-01T10:31:02.0435509Z 
2022-12-01T10:31:02.0435580Z OK (skipped=1)
2022-12-01T10:31:02.0435585Z 
2022-12-01T10:31:02.0435670Z Generating XML reports...
2022-12-01T10:31:02.0436011Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102428.xml
2022-12-01T10:31:02.0436246Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.0436520Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0436646Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0436928Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0437050Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0437069Z 
2022-12-01T10:31:02.0437128Z Running tests...
2022-12-01T10:31:02.0437323Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0437571Z   test_ddp_broadcast_buffer_via_hook (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23005
2022-12-01T10:31:02.0437730Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23006
2022-12-01T10:31:02.0437926Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 23007
2022-12-01T10:31:02.0438224Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0438347Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0438626Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0438748Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0438924Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:31:02.0439193Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0439315Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0439598Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0439733Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0439912Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:31:02.0440181Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0440292Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0440569Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0440703Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0440879Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:31:02.0441179Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0441479Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0441775Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0441941Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:31:02.0442103Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:31:02.0442253Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:31:02.0442359Z skip: Need at least 2 CUDA devices (1.754s)
2022-12-01T10:31:02.0442365Z 
2022-12-01T10:31:02.0442560Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0442636Z Ran 1 test in 1.754s
2022-12-01T10:31:02.0442641Z 
2022-12-01T10:31:02.0442743Z OK (skipped=1)
2022-12-01T10:31:02.0442747Z 
2022-12-01T10:31:02.0442834Z Generating XML reports...
2022-12-01T10:31:02.0443178Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102431.xml
2022-12-01T10:31:02.0443409Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.0443684Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0443797Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0444082Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0444219Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0444223Z 
2022-12-01T10:31:02.0444296Z Running tests...
2022-12-01T10:31:02.0444491Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0445252Z   test_ddp_buffer_hook_allreduce (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78641 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.236s)
2022-12-01T10:31:02.0445293Z 
2022-12-01T10:31:02.0445593Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0445703Z Ran 1 test in 0.236s
2022-12-01T10:31:02.0445711Z 
2022-12-01T10:31:02.0445809Z OK (skipped=1)
2022-12-01T10:31:02.0445814Z 
2022-12-01T10:31:02.0445900Z Generating XML reports...
2022-12-01T10:31:02.0446230Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102434.xml
2022-12-01T10:31:02.0446468Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.0446746Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0446873Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0447157Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0447293Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0447299Z 
2022-12-01T10:31:02.0447372Z Running tests...
2022-12-01T10:31:02.0447567Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0448289Z   test_ddp_buffer_hook_allreduce_return_future (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77261 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.236s)
2022-12-01T10:31:02.0448296Z 
2022-12-01T10:31:02.0448493Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0448555Z Ran 1 test in 0.236s
2022-12-01T10:31:02.0448560Z 
2022-12-01T10:31:02.0448632Z OK (skipped=1)
2022-12-01T10:31:02.0448636Z 
2022-12-01T10:31:02.0448724Z Generating XML reports...
2022-12-01T10:31:02.0449064Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102436.xml
2022-12-01T10:31:02.0449297Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.0449572Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0449695Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0449980Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0450101Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0450123Z 
2022-12-01T10:31:02.0450185Z Running tests...
2022-12-01T10:31:02.0450379Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0450713Z   test_ddp_build_debug_param_to_name_mapping (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'nccl', 'gloo'} (0.003s)
2022-12-01T10:31:02.0450718Z 
2022-12-01T10:31:02.0450911Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0450987Z Ran 1 test in 0.003s
2022-12-01T10:31:02.0450991Z 
2022-12-01T10:31:02.0451063Z OK (skipped=1)
2022-12-01T10:31:02.0451067Z 
2022-12-01T10:31:02.0451152Z Generating XML reports...
2022-12-01T10:31:02.0451489Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102437.xml
2022-12-01T10:31:02.0451751Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.0452054Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0452178Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0452460Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0452596Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0452601Z 
2022-12-01T10:31:02.0452674Z Running tests...
2022-12-01T10:31:02.0452866Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0453130Z   test_ddp_build_debug_param_to_name_mapping_requires_grad (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23088
2022-12-01T10:31:02.0453290Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23089
2022-12-01T10:31:02.0453433Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 23090
2022-12-01T10:31:02.0453711Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0453835Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0454116Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0454251Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0454431Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:31:02.0454704Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0454827Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0455110Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0455311Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0455496Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:31:02.0455770Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0455892Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0456169Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0456304Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0456481Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:31:02.0456783Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0457069Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0457360Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0457527Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:31:02.0457693Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:31:02.0457856Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:31:02.0457962Z skip: Need at least 2 CUDA devices (1.857s)
2022-12-01T10:31:02.0457967Z 
2022-12-01T10:31:02.0458167Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0458243Z Ran 1 test in 1.857s
2022-12-01T10:31:02.0458284Z 
2022-12-01T10:31:02.0458358Z OK (skipped=1)
2022-12-01T10:31:02.0458400Z 
2022-12-01T10:31:02.0458472Z Generating XML reports...
2022-12-01T10:31:02.0458814Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102439.xml
2022-12-01T10:31:02.0459046Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.0459319Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0459443Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0459722Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0459858Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0459863Z 
2022-12-01T10:31:02.0459936Z Running tests...
2022-12-01T10:31:02.0460132Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0460460Z   test_ddp_comm_hook_logging (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23141
2022-12-01T10:31:02.0460620Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23142
2022-12-01T10:31:02.0460775Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 23143
2022-12-01T10:31:02.0461049Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0461174Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0461456Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0461593Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0461775Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:31:02.0462033Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0462157Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0462437Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0462572Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0462751Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:31:02.0463021Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0463145Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0463423Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0463559Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0463723Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:31:02.0464022Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0464318Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0464612Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0464777Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:31:02.0464937Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:31:02.0465155Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:31:02.0465295Z skip: Need at least 3 CUDA devices (1.855s)
2022-12-01T10:31:02.0465301Z 
2022-12-01T10:31:02.0465499Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0465561Z Ran 1 test in 1.855s
2022-12-01T10:31:02.0465566Z 
2022-12-01T10:31:02.0465640Z OK (skipped=1)
2022-12-01T10:31:02.0465645Z 
2022-12-01T10:31:02.0465731Z Generating XML reports...
2022-12-01T10:31:02.0466071Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102442.xml
2022-12-01T10:31:02.0466303Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.0466580Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0466706Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0466990Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0467114Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0467133Z 
2022-12-01T10:31:02.0467193Z Running tests...
2022-12-01T10:31:02.0467390Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0467736Z   test_ddp_control_flow_different_across_ranks (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'gloo', 'nccl'} (0.004s)
2022-12-01T10:31:02.0467741Z 
2022-12-01T10:31:02.0467934Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0468010Z Ran 1 test in 0.004s
2022-12-01T10:31:02.0468015Z 
2022-12-01T10:31:02.0468087Z OK (skipped=1)
2022-12-01T10:31:02.0468092Z 
2022-12-01T10:31:02.0468177Z Generating XML reports...
2022-12-01T10:31:02.0468520Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102445.xml
2022-12-01T10:31:02.0468742Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.0469017Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0469143Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0469425Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0469564Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0469569Z 
2022-12-01T10:31:02.0469642Z Running tests...
2022-12-01T10:31:02.0469839Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0470172Z   test_ddp_control_flow_same_across_ranks (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'gloo', 'nccl'} (0.003s)
2022-12-01T10:31:02.0470178Z 
2022-12-01T10:31:02.0470372Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0470436Z Ran 1 test in 0.003s
2022-12-01T10:31:02.0470454Z 
2022-12-01T10:31:02.0470512Z OK (skipped=1)
2022-12-01T10:31:02.0470516Z 
2022-12-01T10:31:02.0470600Z Generating XML reports...
2022-12-01T10:31:02.0470938Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102447.xml
2022-12-01T10:31:02.0471167Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.0471441Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0471567Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0471849Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0472021Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0472052Z 
2022-12-01T10:31:02.0472111Z Running tests...
2022-12-01T10:31:02.0472307Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0472540Z   test_ddp_create_graph (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23214
2022-12-01T10:31:02.0472700Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23215
2022-12-01T10:31:02.0472855Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 23216
2022-12-01T10:31:02.0473128Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0473252Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0473531Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0473669Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0473834Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:31:02.0474109Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0474231Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0474511Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0474645Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0474822Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:31:02.0475091Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0475216Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0475481Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0475619Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0475796Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:31:02.0476091Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0476388Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0476683Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0476850Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:31:02.0477039Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpejasvuyh
2022-12-01T10:31:02.0477239Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpejasvuyh/_remote_module_non_scriptable.py
2022-12-01T10:31:02.0477390Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:31:02.0477553Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:31:02.0477737Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp85xytxcd
2022-12-01T10:31:02.0477919Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2t8dkdaa
2022-12-01T10:31:02.0478114Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp85xytxcd/_remote_module_non_scriptable.py
2022-12-01T10:31:02.0478307Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2t8dkdaa/_remote_module_non_scriptable.py
2022-12-01T10:31:02.0479033Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-12-01T10:31:02.0479756Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-12-01T10:31:02.0480441Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-12-01T10:31:02.0481340Z /opt/conda/lib/python3.7/site-packages/torch/autograd/__init__.py:199: UserWarning: Using backward() with create_graph=True will create a reference cycle between the parameter and its gradient which can cause a memory leak. We recommend using autograd.grad when creating the graph to avoid this. If you have to use this function, make sure to reset the .grad fields of your parameters to None after use to break the cycle and avoid the leak. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/engine.cpp:1064.)
2022-12-01T10:31:02.0481513Z   allow_unreachable=True, accumulate_grad=True)  # Calls into the C++ engine to run the backward pass
2022-12-01T10:31:02.0481685Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:31:02.0482573Z /opt/conda/lib/python3.7/site-packages/torch/autograd/__init__.py:199: UserWarning: Using backward() with create_graph=True will create a reference cycle between the parameter and its gradient which can cause a memory leak. We recommend using autograd.grad when creating the graph to avoid this. If you have to use this function, make sure to reset the .grad fields of your parameters to None after use to break the cycle and avoid the leak. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/engine.cpp:1064.)
2022-12-01T10:31:02.0482745Z   allow_unreachable=True, accumulate_grad=True)  # Calls into the C++ engine to run the backward pass
2022-12-01T10:31:02.0482914Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:31:02.0483798Z /opt/conda/lib/python3.7/site-packages/torch/autograd/__init__.py:199: UserWarning: Using backward() with create_graph=True will create a reference cycle between the parameter and its gradient which can cause a memory leak. We recommend using autograd.grad when creating the graph to avoid this. If you have to use this function, make sure to reset the .grad fields of your parameters to None after use to break the cycle and avoid the leak. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/engine.cpp:1064.)
2022-12-01T10:31:02.0483967Z   allow_unreachable=True, accumulate_grad=True)  # Calls into the C++ engine to run the backward pass
2022-12-01T10:31:02.0484133Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:31:02.0484807Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-12-01T10:31:02.0485528Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-12-01T10:31:02.0486239Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-12-01T10:31:02.0486915Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-12-01T10:31:02.0487591Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-12-01T10:31:02.0488270Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-12-01T10:31:02.0488946Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-12-01T10:31:02.0489619Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-12-01T10:31:02.0490294Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-12-01T10:31:02.0490968Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-12-01T10:31:02.0491640Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-12-01T10:31:02.0492335Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-12-01T10:31:02.0493026Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-12-01T10:31:02.0493691Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-12-01T10:31:02.0494361Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-12-01T10:31:02.0494433Z ok (1.852s)
2022-12-01T10:31:02.0494440Z 
2022-12-01T10:31:02.0494641Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0494704Z Ran 1 test in 1.852s
2022-12-01T10:31:02.0494709Z 
2022-12-01T10:31:02.0494771Z OK
2022-12-01T10:31:02.0494776Z 
2022-12-01T10:31:02.0494861Z Generating XML reports...
2022-12-01T10:31:02.0495203Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102448.xml
2022-12-01T10:31:02.0495517Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.0495798Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0495924Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0496209Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0496330Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0496349Z 
2022-12-01T10:31:02.0496408Z Running tests...
2022-12-01T10:31:02.0496603Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0496898Z   test_ddp_device (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'nccl', 'gloo'} (0.004s)
2022-12-01T10:31:02.0496903Z 
2022-12-01T10:31:02.0497097Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0497174Z Ran 1 test in 0.004s
2022-12-01T10:31:02.0497179Z 
2022-12-01T10:31:02.0497251Z OK (skipped=1)
2022-12-01T10:31:02.0497255Z 
2022-12-01T10:31:02.0497343Z Generating XML reports...
2022-12-01T10:31:02.0497688Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102451.xml
2022-12-01T10:31:02.0497909Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.0498186Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0498309Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0498591Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0498726Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0498731Z 
2022-12-01T10:31:02.0498805Z Running tests...
2022-12-01T10:31:02.0499000Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0499352Z   test_ddp_forward_backward_hook (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'gloo', 'nccl'} (0.003s)
2022-12-01T10:31:02.0499382Z 
2022-12-01T10:31:02.0499577Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0499639Z Ran 1 test in 0.003s
2022-12-01T10:31:02.0499643Z 
2022-12-01T10:31:02.0499715Z OK (skipped=1)
2022-12-01T10:31:02.0499720Z 
2022-12-01T10:31:02.0499805Z Generating XML reports...
2022-12-01T10:31:02.0500141Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102453.xml
2022-12-01T10:31:02.0500512Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.0500789Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0500915Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0501199Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0501337Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0501342Z 
2022-12-01T10:31:02.0501402Z Running tests...
2022-12-01T10:31:02.0501596Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0502284Z   test_ddp_grad_div_uneven_inputs (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78685 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.237s)
2022-12-01T10:31:02.0502289Z 
2022-12-01T10:31:02.0502487Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0502564Z Ran 1 test in 0.237s
2022-12-01T10:31:02.0502568Z 
2022-12-01T10:31:02.0502640Z OK (skipped=1)
2022-12-01T10:31:02.0502647Z 
2022-12-01T10:31:02.0502733Z Generating XML reports...
2022-12-01T10:31:02.0503073Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102454.xml
2022-12-01T10:31:02.0503308Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.0503583Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0503693Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0503972Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0504107Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0504112Z 
2022-12-01T10:31:02.0504184Z Running tests...
2022-12-01T10:31:02.0504379Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0505068Z   test_ddp_hook_parity_allreduce (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77293 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.237s)
2022-12-01T10:31:02.0505076Z 
2022-12-01T10:31:02.0505270Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0505344Z Ran 1 test in 0.238s
2022-12-01T10:31:02.0505349Z 
2022-12-01T10:31:02.0505424Z OK (skipped=1)
2022-12-01T10:31:02.0505428Z 
2022-12-01T10:31:02.0505512Z Generating XML reports...
2022-12-01T10:31:02.0505834Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102456.xml
2022-12-01T10:31:02.0506120Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.0506394Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0506565Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0506849Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0506984Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0506989Z 
2022-12-01T10:31:02.0507061Z Running tests...
2022-12-01T10:31:02.0507256Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0507498Z   test_ddp_hook_parity_allreduce_process_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23307
2022-12-01T10:31:02.0507658Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23308
2022-12-01T10:31:02.0507814Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 23309
2022-12-01T10:31:02.0508091Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0508216Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0508500Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0508636Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0508814Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:31:02.0509086Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0509195Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0509477Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0509615Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0509796Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:31:02.0510068Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0510191Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0510470Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0510604Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0510768Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:31:02.0511069Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0511364Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0511661Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0511831Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:31:02.0511996Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:31:02.0512160Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:31:02.0512265Z skip: Need at least 3 CUDA devices (1.766s)
2022-12-01T10:31:02.0512270Z 
2022-12-01T10:31:02.0512468Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0512529Z Ran 1 test in 1.766s
2022-12-01T10:31:02.0512547Z 
2022-12-01T10:31:02.0512606Z OK (skipped=1)
2022-12-01T10:31:02.0512649Z 
2022-12-01T10:31:02.0512736Z Generating XML reports...
2022-12-01T10:31:02.0513100Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102457.xml
2022-12-01T10:31:02.0513332Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.0513604Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0513730Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0514008Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0514145Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0514150Z 
2022-12-01T10:31:02.0514210Z Running tests...
2022-12-01T10:31:02.0514404Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0514654Z   test_ddp_hook_parity_post_localSGD (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23360
2022-12-01T10:31:02.0514814Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23361
2022-12-01T10:31:02.0514970Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 23362
2022-12-01T10:31:02.0515243Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0515366Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0515645Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0515780Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0515943Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:31:02.0516217Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0516340Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0516621Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0516755Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0516930Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:31:02.0517204Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0517326Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0517591Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0517727Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0517906Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:31:02.0518204Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0518501Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0518788Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0518953Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:31:02.0519113Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:31:02.0519277Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:31:02.0519400Z skip: Need at least 3 CUDA devices (1.861s)
2022-12-01T10:31:02.0519447Z 
2022-12-01T10:31:02.0519630Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0519706Z Ran 1 test in 1.862s
2022-12-01T10:31:02.0519711Z 
2022-12-01T10:31:02.0519782Z OK (skipped=1)
2022-12-01T10:31:02.0519786Z 
2022-12-01T10:31:02.0519871Z Generating XML reports...
2022-12-01T10:31:02.0520211Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102501.xml
2022-12-01T10:31:02.0520446Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.0520721Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0520846Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0521112Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0521252Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0521258Z 
2022-12-01T10:31:02.0521332Z Running tests...
2022-12-01T10:31:02.0521527Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0522214Z   test_ddp_hook_parity_powerSGD (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77378 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.239s)
2022-12-01T10:31:02.0522221Z 
2022-12-01T10:31:02.0522413Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0522488Z Ran 1 test in 0.239s
2022-12-01T10:31:02.0522493Z 
2022-12-01T10:31:02.0522567Z OK (skipped=1)
2022-12-01T10:31:02.0522571Z 
2022-12-01T10:31:02.0522658Z Generating XML reports...
2022-12-01T10:31:02.0522995Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102504.xml
2022-12-01T10:31:02.0523218Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.0523492Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0523616Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0523957Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0524089Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0524094Z 
2022-12-01T10:31:02.0524168Z Running tests...
2022-12-01T10:31:02.0524364Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0524607Z   test_ddp_hook_pickling_powerSGD (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23423
2022-12-01T10:31:02.0524754Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23424
2022-12-01T10:31:02.0524908Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 23425
2022-12-01T10:31:02.0525184Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0525308Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0525586Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0525721Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0525899Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:31:02.0526201Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0526349Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0526613Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0526747Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0526924Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:31:02.0527197Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0527320Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0527599Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0527735Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0527909Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:31:02.0528213Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0528497Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0528793Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0528996Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:31:02.0529162Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:31:02.0529325Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:31:02.0529432Z skip: Need at least 3 CUDA devices (1.760s)
2022-12-01T10:31:02.0529438Z 
2022-12-01T10:31:02.0529639Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0529716Z Ran 1 test in 1.760s
2022-12-01T10:31:02.0529721Z 
2022-12-01T10:31:02.0529778Z OK (skipped=1)
2022-12-01T10:31:02.0529796Z 
2022-12-01T10:31:02.0529867Z Generating XML reports...
2022-12-01T10:31:02.0530207Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102506.xml
2022-12-01T10:31:02.0530438Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.0530710Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0530835Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0531113Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0531251Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0531258Z 
2022-12-01T10:31:02.0531331Z Running tests...
2022-12-01T10:31:02.0531511Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0531783Z   test_ddp_hook_with_optimizer_parity_adam_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23476
2022-12-01T10:31:02.0531941Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23477
2022-12-01T10:31:02.0532096Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 23478
2022-12-01T10:31:02.0532367Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0532492Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0532810Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0532972Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0533149Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:31:02.0533434Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0533703Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0533826Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0534103Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0534238Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0534418Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:31:02.0534692Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0534817Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0535097Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0535217Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0535473Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:31:02.0535778Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0536072Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0536242Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:31:02.0536411Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:31:02.0536571Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:31:02.0536677Z skip: Need at least 2 CUDA devices (1.856s)
2022-12-01T10:31:02.0536683Z 
2022-12-01T10:31:02.0536881Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0536943Z Ran 1 test in 1.856s
2022-12-01T10:31:02.0536948Z 
2022-12-01T10:31:02.0537021Z OK (skipped=1)
2022-12-01T10:31:02.0537025Z 
2022-12-01T10:31:02.0537110Z Generating XML reports...
2022-12-01T10:31:02.0537456Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102509.xml
2022-12-01T10:31:02.0537688Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.0537965Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0538091Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0538371Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0538491Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0538510Z 
2022-12-01T10:31:02.0538570Z Running tests...
2022-12-01T10:31:02.0538766Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0539038Z   test_ddp_hook_with_optimizer_parity_adam_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23529
2022-12-01T10:31:02.0539195Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23530
2022-12-01T10:31:02.0539412Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 23531
2022-12-01T10:31:02.0539688Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0539842Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0540126Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0540248Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0540646Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:31:02.0540923Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0541046Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0541331Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0541469Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0541649Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:31:02.0541948Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0542219Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0542329Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0542607Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0542744Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0542923Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:31:02.0543222Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0543516Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0543684Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:31:02.0543848Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:31:02.0543998Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:31:02.0544103Z skip: Need at least 2 CUDA devices (1.754s)
2022-12-01T10:31:02.0544109Z 
2022-12-01T10:31:02.0544303Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0544379Z Ran 1 test in 1.754s
2022-12-01T10:31:02.0544384Z 
2022-12-01T10:31:02.0544456Z OK (skipped=1)
2022-12-01T10:31:02.0544460Z 
2022-12-01T10:31:02.0544547Z Generating XML reports...
2022-12-01T10:31:02.0544885Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102512.xml
2022-12-01T10:31:02.0545119Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.0545394Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0545506Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0545788Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0545923Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0545928Z 
2022-12-01T10:31:02.0546002Z Running tests...
2022-12-01T10:31:02.0546196Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0546566Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23582
2022-12-01T10:31:02.0546757Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23583
2022-12-01T10:31:02.0546915Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 23584
2022-12-01T10:31:02.0547189Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0547301Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0547579Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0547715Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0547897Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:31:02.0548169Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0548295Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0548572Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0548708Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0548872Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:31:02.0549144Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0549266Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0549545Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0549684Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0549865Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:31:02.0550162Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0550460Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0550751Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0550915Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:31:02.0551066Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:31:02.0551231Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:31:02.0551336Z skip: Need at least 2 CUDA devices (1.853s)
2022-12-01T10:31:02.0551343Z 
2022-12-01T10:31:02.0551541Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0551621Z Ran 1 test in 1.853s
2022-12-01T10:31:02.0551626Z 
2022-12-01T10:31:02.0551698Z OK (skipped=1)
2022-12-01T10:31:02.0551703Z 
2022-12-01T10:31:02.0551788Z Generating XML reports...
2022-12-01T10:31:02.0552125Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102515.xml
2022-12-01T10:31:02.0552344Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.0552618Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0552742Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0553053Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0553229Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0553234Z 
2022-12-01T10:31:02.0553307Z Running tests...
2022-12-01T10:31:02.0553503Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0553819Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23635
2022-12-01T10:31:02.0553978Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23636
2022-12-01T10:31:02.0554118Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 23637
2022-12-01T10:31:02.0554391Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0554514Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0554797Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0554935Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0555114Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:31:02.0555385Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0555508Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0555789Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0555910Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0556090Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:31:02.0556359Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0556483Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0556760Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0556894Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0557069Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:31:02.0557368Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0557650Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0557941Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0558110Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:31:02.0558273Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:31:02.0558437Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:31:02.0558543Z skip: Need at least 2 CUDA devices (1.753s)
2022-12-01T10:31:02.0558548Z 
2022-12-01T10:31:02.0558746Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0558822Z Ran 1 test in 1.754s
2022-12-01T10:31:02.0558827Z 
2022-12-01T10:31:02.0558899Z OK (skipped=1)
2022-12-01T10:31:02.0558903Z 
2022-12-01T10:31:02.0558974Z Generating XML reports...
2022-12-01T10:31:02.0559312Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102519.xml
2022-12-01T10:31:02.0559573Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.0559878Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0560002Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0560283Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0560418Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0560423Z 
2022-12-01T10:31:02.0560497Z Running tests...
2022-12-01T10:31:02.0560691Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0560997Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23688
2022-12-01T10:31:02.0561157Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23689
2022-12-01T10:31:02.0561315Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 23690
2022-12-01T10:31:02.0561587Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0561711Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0561991Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0562128Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0562307Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:31:02.0562579Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0562691Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0562969Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0563107Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0563284Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:31:02.0563552Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0563676Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0563952Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0564087Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0564249Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:31:02.0564550Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0564847Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0565134Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0565299Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:31:02.0565465Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:31:02.0565629Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:31:02.0565733Z skip: Need at least 2 CUDA devices (1.854s)
2022-12-01T10:31:02.0565738Z 
2022-12-01T10:31:02.0565931Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0566022Z Ran 1 test in 1.855s
2022-12-01T10:31:02.0566066Z 
2022-12-01T10:31:02.0566126Z OK (skipped=1)
2022-12-01T10:31:02.0566131Z 
2022-12-01T10:31:02.0566217Z Generating XML reports...
2022-12-01T10:31:02.0566557Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102522.xml
2022-12-01T10:31:02.0566788Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.0567062Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0567185Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0567462Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0567599Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0567603Z 
2022-12-01T10:31:02.0567664Z Running tests...
2022-12-01T10:31:02.0567858Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0568171Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23741
2022-12-01T10:31:02.0568329Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23742
2022-12-01T10:31:02.0568483Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 23743
2022-12-01T10:31:02.0568752Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0568875Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0569151Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0569288Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0569453Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:31:02.0569728Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0569852Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0570130Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0570266Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0570444Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:31:02.0570717Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0570845Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0571110Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0571248Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0571428Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:31:02.0571729Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0572027Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0572319Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0572485Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:31:02.0572676Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:31:02.0572865Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:31:02.0572956Z skip: Need at least 2 CUDA devices (1.856s)
2022-12-01T10:31:02.0572975Z 
2022-12-01T10:31:02.0573157Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0573232Z Ran 1 test in 1.856s
2022-12-01T10:31:02.0573238Z 
2022-12-01T10:31:02.0573309Z OK (skipped=1)
2022-12-01T10:31:02.0573314Z 
2022-12-01T10:31:02.0573398Z Generating XML reports...
2022-12-01T10:31:02.0573737Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102525.xml
2022-12-01T10:31:02.0573968Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.0574248Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0574373Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0574644Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0574781Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0574786Z 
2022-12-01T10:31:02.0574859Z Running tests...
2022-12-01T10:31:02.0575054Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0575442Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23794
2022-12-01T10:31:02.0575604Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23795
2022-12-01T10:31:02.0575761Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 23796
2022-12-01T10:31:02.0576043Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0576170Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0576437Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0576572Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0576750Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:31:02.0577020Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0577143Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0577425Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0577565Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0577744Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:31:02.0578015Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0578126Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0578403Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0578538Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0578714Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:31:02.0579015Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0579345Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0579662Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0579828Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:31:02.0579993Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:31:02.0580141Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:31:02.0580245Z skip: Need at least 2 CUDA devices (1.852s)
2022-12-01T10:31:02.0580252Z 
2022-12-01T10:31:02.0580561Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0580638Z Ran 1 test in 1.853s
2022-12-01T10:31:02.0580644Z 
2022-12-01T10:31:02.0580717Z OK (skipped=1)
2022-12-01T10:31:02.0580722Z 
2022-12-01T10:31:02.0580806Z Generating XML reports...
2022-12-01T10:31:02.0581151Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102528.xml
2022-12-01T10:31:02.0581390Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.0581653Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0581778Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0582059Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0582194Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0582198Z 
2022-12-01T10:31:02.0582271Z Running tests...
2022-12-01T10:31:02.0582467Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0582782Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23847
2022-12-01T10:31:02.0582943Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23848
2022-12-01T10:31:02.0583098Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 23849
2022-12-01T10:31:02.0583359Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0583483Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0583764Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0583898Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0584074Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:31:02.0584349Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0584476Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0584756Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0584891Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0585055Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:31:02.0585326Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0585449Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0585730Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0585915Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0586127Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:31:02.0586426Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0586722Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0587013Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0587165Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:31:02.0587330Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:31:02.0587492Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:31:02.0587599Z skip: Need at least 2 CUDA devices (1.854s)
2022-12-01T10:31:02.0587608Z 
2022-12-01T10:31:02.0587804Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0587881Z Ran 1 test in 1.854s
2022-12-01T10:31:02.0587886Z 
2022-12-01T10:31:02.0587957Z OK (skipped=1)
2022-12-01T10:31:02.0587962Z 
2022-12-01T10:31:02.0588047Z Generating XML reports...
2022-12-01T10:31:02.0588369Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102532.xml
2022-12-01T10:31:02.0588605Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.0588878Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0589003Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0589285Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0589421Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0589428Z 
2022-12-01T10:31:02.0589501Z Running tests...
2022-12-01T10:31:02.0589697Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0590015Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23900
2022-12-01T10:31:02.0590159Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23901
2022-12-01T10:31:02.0590316Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 23902
2022-12-01T10:31:02.0590589Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0590715Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0590994Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0591133Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0591310Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:31:02.0591580Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0591702Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0591967Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0592103Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0592280Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:31:02.0592607Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0592920Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0593042Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0593322Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0593458Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0593631Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:31:02.0593913Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0594215Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0594379Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:31:02.0594542Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:31:02.0594704Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:31:02.0594808Z skip: Need at least 2 CUDA devices (1.855s)
2022-12-01T10:31:02.0594814Z 
2022-12-01T10:31:02.0595011Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0595086Z Ran 1 test in 1.855s
2022-12-01T10:31:02.0595091Z 
2022-12-01T10:31:02.0595149Z OK (skipped=1)
2022-12-01T10:31:02.0595168Z 
2022-12-01T10:31:02.0595283Z Generating XML reports...
2022-12-01T10:31:02.0595623Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102535.xml
2022-12-01T10:31:02.0595858Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.0596132Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0596259Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0596540Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0596677Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0596682Z 
2022-12-01T10:31:02.0596755Z Running tests...
2022-12-01T10:31:02.0596936Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0597251Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23953
2022-12-01T10:31:02.0597411Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23954
2022-12-01T10:31:02.0597567Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 23955
2022-12-01T10:31:02.0597845Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0597968Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0598250Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0598386Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0598564Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:31:02.0598821Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0598945Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0599252Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0599427Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0599605Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:31:02.0599878Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0600002Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0600284Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0600417Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0600579Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:31:02.0600878Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0601179Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0601472Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0601638Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:31:02.0601798Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:31:02.0601959Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:31:02.0602063Z skip: Need at least 2 CUDA devices (1.854s)
2022-12-01T10:31:02.0602069Z 
2022-12-01T10:31:02.0602265Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0602327Z Ran 1 test in 1.854s
2022-12-01T10:31:02.0602334Z 
2022-12-01T10:31:02.0602406Z OK (skipped=1)
2022-12-01T10:31:02.0602413Z 
2022-12-01T10:31:02.0602498Z Generating XML reports...
2022-12-01T10:31:02.0602840Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102538.xml
2022-12-01T10:31:02.0603073Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.0603347Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0603472Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0603750Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0603872Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0603891Z 
2022-12-01T10:31:02.0603951Z Running tests...
2022-12-01T10:31:02.0604150Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0604474Z   test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24006
2022-12-01T10:31:02.0604633Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24007
2022-12-01T10:31:02.0604788Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 24008
2022-12-01T10:31:02.0605059Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0605183Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0605464Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0605586Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0605792Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:31:02.0606090Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0606214Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0606493Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0606630Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0606805Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:31:02.0607073Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0607182Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0607555Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0607721Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0607896Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:31:02.0608199Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0608494Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0608785Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0608949Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:31:02.0609108Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:31:02.0609261Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:31:02.0609369Z skip: Need at least 2 CUDA devices (1.853s)
2022-12-01T10:31:02.0609375Z 
2022-12-01T10:31:02.0609571Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0609647Z Ran 1 test in 1.853s
2022-12-01T10:31:02.0609652Z 
2022-12-01T10:31:02.0609724Z OK (skipped=1)
2022-12-01T10:31:02.0609729Z 
2022-12-01T10:31:02.0609813Z Generating XML reports...
2022-12-01T10:31:02.0610154Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102542.xml
2022-12-01T10:31:02.0610385Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.0610659Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0610768Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0611049Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0611186Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0611191Z 
2022-12-01T10:31:02.0611263Z Running tests...
2022-12-01T10:31:02.0611459Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0611730Z   test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24059
2022-12-01T10:31:02.0611887Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24060
2022-12-01T10:31:02.0612042Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 24061
2022-12-01T10:31:02.0612314Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0612499Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0612807Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0612943Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0613122Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:31:02.0613391Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0613514Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0613795Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0613929Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0614094Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:31:02.0614364Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0614489Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0614766Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0614900Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0615078Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:31:02.0615450Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0615750Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0616046Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0616201Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:31:02.0616359Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:31:02.0616521Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:31:02.0616627Z skip: Need at least 2 CUDA devices (1.852s)
2022-12-01T10:31:02.0616632Z 
2022-12-01T10:31:02.0616826Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0616902Z Ran 1 test in 1.852s
2022-12-01T10:31:02.0616907Z 
2022-12-01T10:31:02.0616979Z OK (skipped=1)
2022-12-01T10:31:02.0616984Z 
2022-12-01T10:31:02.0617069Z Generating XML reports...
2022-12-01T10:31:02.0617411Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102545.xml
2022-12-01T10:31:02.0617629Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.0617907Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0618031Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0618313Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0618450Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0618455Z 
2022-12-01T10:31:02.0618529Z Running tests...
2022-12-01T10:31:02.0618726Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0619040Z   test_ddp_ignore_params_arg (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'gloo', 'nccl'} (0.001s)
2022-12-01T10:31:02.0619045Z 
2022-12-01T10:31:02.0619275Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0619362Z Ran 1 test in 0.002s
2022-12-01T10:31:02.0619367Z 
2022-12-01T10:31:02.0619438Z OK (skipped=1)
2022-12-01T10:31:02.0619443Z 
2022-12-01T10:31:02.0619527Z Generating XML reports...
2022-12-01T10:31:02.0619863Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102548.xml
2022-12-01T10:31:02.0620092Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.0620453Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0620579Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0620861Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0620983Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0621004Z 
2022-12-01T10:31:02.0621065Z Running tests...
2022-12-01T10:31:02.0621265Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0621497Z   test_ddp_inference (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24122
2022-12-01T10:31:02.0621654Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24123
2022-12-01T10:31:02.0621809Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 24124
2022-12-01T10:31:02.0622083Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0622207Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0622487Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0622612Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0622791Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:31:02.0623066Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0623189Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0623469Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0623605Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0623783Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:31:02.0624055Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0624178Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0624444Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0624582Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0624759Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:31:02.0625057Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0625354Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0625653Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0625822Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:31:02.0626042Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:31:02.0626205Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:31:02.0626330Z skip: Need at least 2 CUDA devices (1.853s)
2022-12-01T10:31:02.0626335Z 
2022-12-01T10:31:02.0626533Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0626609Z Ran 1 test in 1.853s
2022-12-01T10:31:02.0626614Z 
2022-12-01T10:31:02.0626686Z OK (skipped=1)
2022-12-01T10:31:02.0626690Z 
2022-12-01T10:31:02.0626774Z Generating XML reports...
2022-12-01T10:31:02.0627117Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102550.xml
2022-12-01T10:31:02.0627349Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.0627623Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0627736Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0628021Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0628157Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0628162Z 
2022-12-01T10:31:02.0628237Z Running tests...
2022-12-01T10:31:02.0628432Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0628674Z   test_ddp_join_model_equivalence (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24175
2022-12-01T10:31:02.0628831Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24176
2022-12-01T10:31:02.0628986Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 24177
2022-12-01T10:31:02.0629261Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0629370Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0629652Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0629787Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0629965Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:31:02.0630236Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0630383Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0630661Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0630795Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0630959Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:31:02.0631229Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0631351Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0631629Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0631763Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0631939Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:31:02.0632241Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0632537Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0632860Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0633038Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:31:02.0633202Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:31:02.0633365Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:31:02.0633471Z skip: Need at least 2 CUDA devices (1.757s)
2022-12-01T10:31:02.0633476Z 
2022-12-01T10:31:02.0633673Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0633749Z Ran 1 test in 1.757s
2022-12-01T10:31:02.0633754Z 
2022-12-01T10:31:02.0633826Z OK (skipped=1)
2022-12-01T10:31:02.0633830Z 
2022-12-01T10:31:02.0633914Z Generating XML reports...
2022-12-01T10:31:02.0634255Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102553.xml
2022-12-01T10:31:02.0634476Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.0634753Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0634878Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0635160Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0635301Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0635306Z 
2022-12-01T10:31:02.0635378Z Running tests...
2022-12-01T10:31:02.0635573Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0635811Z   test_ddp_logging_data_cpu (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24228
2022-12-01T10:31:02.0635969Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24229
2022-12-01T10:31:02.0636111Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 24230
2022-12-01T10:31:02.0636390Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0636513Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0636793Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0636927Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0637106Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:31:02.0637376Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0637501Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0637768Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0637906Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0638082Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:31:02.0638351Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0638476Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0638755Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0638889Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0639066Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:31:02.0639394Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0639725Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0640017Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0640183Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:31:02.0640342Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:31:02.0640530Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_v0idg_z
2022-12-01T10:31:02.0640715Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw4kx4aq1
2022-12-01T10:31:02.0640917Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_v0idg_z/_remote_module_non_scriptable.py
2022-12-01T10:31:02.0641109Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw4kx4aq1/_remote_module_non_scriptable.py
2022-12-01T10:31:02.0641272Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:31:02.0641442Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq92fdtsy
2022-12-01T10:31:02.0641639Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq92fdtsy/_remote_module_non_scriptable.py
2022-12-01T10:31:02.0641809Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:31:02.0641980Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:31:02.0642145Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:31:02.0642215Z ok (2.061s)
2022-12-01T10:31:02.0642220Z 
2022-12-01T10:31:02.0642418Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0642493Z Ran 1 test in 2.061s
2022-12-01T10:31:02.0642500Z 
2022-12-01T10:31:02.0642548Z OK
2022-12-01T10:31:02.0642553Z 
2022-12-01T10:31:02.0642638Z Generating XML reports...
2022-12-01T10:31:02.0642976Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102556.xml
2022-12-01T10:31:02.0643207Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.0643483Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0643607Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0643888Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0644024Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0644029Z 
2022-12-01T10:31:02.0644103Z Running tests...
2022-12-01T10:31:02.0644284Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0644523Z   test_ddp_logging_data_gpu (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24290
2022-12-01T10:31:02.0644680Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24291
2022-12-01T10:31:02.0644837Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 24292
2022-12-01T10:31:02.0645110Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0645234Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0645514Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0645648Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0645840Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:31:02.0646151Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0646274Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0646556Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0646689Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0646865Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:31:02.0647137Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0647266Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0647549Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0647673Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0647848Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:31:02.0648147Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0648446Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0648739Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0648905Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:31:02.0649072Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:31:02.0649236Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:31:02.0649335Z skip: CUDA is not available. (1.855s)
2022-12-01T10:31:02.0649340Z 
2022-12-01T10:31:02.0649520Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0649598Z Ran 1 test in 1.855s
2022-12-01T10:31:02.0649603Z 
2022-12-01T10:31:02.0649675Z OK (skipped=1)
2022-12-01T10:31:02.0649679Z 
2022-12-01T10:31:02.0649763Z Generating XML reports...
2022-12-01T10:31:02.0650102Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102600.xml
2022-12-01T10:31:02.0650333Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.0650607Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0650733Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0650998Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0651136Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0651141Z 
2022-12-01T10:31:02.0651216Z Running tests...
2022-12-01T10:31:02.0651412Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0651747Z   test_ddp_model_diff_num_params_across_ranks (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'gloo', 'nccl'} (0.002s)
2022-12-01T10:31:02.0651753Z 
2022-12-01T10:31:02.0651950Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0652026Z Ran 1 test in 0.002s
2022-12-01T10:31:02.0652030Z 
2022-12-01T10:31:02.0652102Z OK (skipped=1)
2022-12-01T10:31:02.0652106Z 
2022-12-01T10:31:02.0652190Z Generating XML reports...
2022-12-01T10:31:02.0652545Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102603.xml
2022-12-01T10:31:02.0652805Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.0653079Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0653202Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0653479Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0653615Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0653619Z 
2022-12-01T10:31:02.0653692Z Running tests...
2022-12-01T10:31:02.0653885Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0654213Z   test_ddp_model_diff_shape_across_ranks (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'gloo', 'nccl'} (0.002s)
2022-12-01T10:31:02.0654220Z 
2022-12-01T10:31:02.0654398Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0654473Z Ran 1 test in 0.002s
2022-12-01T10:31:02.0654478Z 
2022-12-01T10:31:02.0654549Z OK (skipped=1)
2022-12-01T10:31:02.0654553Z 
2022-12-01T10:31:02.0654637Z Generating XML reports...
2022-12-01T10:31:02.0654973Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102604.xml
2022-12-01T10:31:02.0655205Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.0655571Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0655695Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0655975Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0656097Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0656104Z 
2022-12-01T10:31:02.0656179Z Running tests...
2022-12-01T10:31:02.0656375Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0656737Z   test_ddp_multiple_nested_unused_params_err_ignore_params (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'nccl', 'gloo'} (0.001s)
2022-12-01T10:31:02.0656743Z 
2022-12-01T10:31:02.0656935Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0657011Z Ran 1 test in 0.002s
2022-12-01T10:31:02.0657016Z 
2022-12-01T10:31:02.0657088Z OK (skipped=1)
2022-12-01T10:31:02.0657092Z 
2022-12-01T10:31:02.0657176Z Generating XML reports...
2022-12-01T10:31:02.0657510Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102606.xml
2022-12-01T10:31:02.0657728Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.0658003Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0658127Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0658404Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0658539Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0658543Z 
2022-12-01T10:31:02.0658616Z Running tests...
2022-12-01T10:31:02.0658811Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0659152Z   test_ddp_multiple_nested_unused_params_error (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'nccl', 'gloo'} (0.001s)
2022-12-01T10:31:02.0659157Z 
2022-12-01T10:31:02.0659388Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0659475Z Ran 1 test in 0.002s
2022-12-01T10:31:02.0659480Z 
2022-12-01T10:31:02.0659553Z OK (skipped=1)
2022-12-01T10:31:02.0659557Z 
2022-12-01T10:31:02.0659642Z Generating XML reports...
2022-12-01T10:31:02.0659979Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102607.xml
2022-12-01T10:31:02.0660212Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.0660592Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0660718Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0661003Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0661125Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0661146Z 
2022-12-01T10:31:02.0661205Z Running tests...
2022-12-01T10:31:02.0661402Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0661708Z   test_ddp_namedtuple (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'nccl', 'gloo'} (0.002s)
2022-12-01T10:31:02.0661713Z 
2022-12-01T10:31:02.0661903Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0661979Z Ran 1 test in 0.002s
2022-12-01T10:31:02.0661984Z 
2022-12-01T10:31:02.0662056Z OK (skipped=1)
2022-12-01T10:31:02.0662061Z 
2022-12-01T10:31:02.0662144Z Generating XML reports...
2022-12-01T10:31:02.0662479Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102608.xml
2022-12-01T10:31:02.0662697Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.0662973Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0663097Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0663380Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0663516Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0663521Z 
2022-12-01T10:31:02.0663595Z Running tests...
2022-12-01T10:31:02.0663788Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0664020Z   test_ddp_new_tensor_in_fwd (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24393
2022-12-01T10:31:02.0664178Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24394
2022-12-01T10:31:02.0664319Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 24395
2022-12-01T10:31:02.0664595Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0664719Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0664998Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0665133Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0665310Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:31:02.0665579Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0665701Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0665966Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0666149Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0666330Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:31:02.0666642Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0666767Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0667045Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0667181Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0667359Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:31:02.0667657Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0667946Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0668241Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0668407Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:31:02.0668568Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:31:02.0668732Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:31:02.0668836Z skip: Need at least 2 CUDA devices (1.858s)
2022-12-01T10:31:02.0668842Z 
2022-12-01T10:31:02.0669039Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0669115Z Ran 1 test in 1.858s
2022-12-01T10:31:02.0669120Z 
2022-12-01T10:31:02.0669191Z OK (skipped=1)
2022-12-01T10:31:02.0669195Z 
2022-12-01T10:31:02.0669266Z Generating XML reports...
2022-12-01T10:31:02.0669611Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102610.xml
2022-12-01T10:31:02.0669844Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.0670115Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0670239Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0670516Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0670651Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0670656Z 
2022-12-01T10:31:02.0670730Z Running tests...
2022-12-01T10:31:02.0670927Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0671628Z   test_ddp_new_tensor_in_fwd_static_graph (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78338 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.234s)
2022-12-01T10:31:02.0671649Z 
2022-12-01T10:31:02.0671830Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0671905Z Ran 1 test in 0.234s
2022-12-01T10:31:02.0671910Z 
2022-12-01T10:31:02.0671980Z OK (skipped=1)
2022-12-01T10:31:02.0671985Z 
2022-12-01T10:31:02.0672068Z Generating XML reports...
2022-12-01T10:31:02.0672405Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102613.xml
2022-12-01T10:31:02.0672639Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.0672942Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0673066Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0673373Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0673494Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0673499Z 
2022-12-01T10:31:02.0673572Z Running tests...
2022-12-01T10:31:02.0673767Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0674099Z   test_ddp_profiling_autograd_profiler (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'gloo', 'nccl'} (0.001s)
2022-12-01T10:31:02.0674104Z 
2022-12-01T10:31:02.0674294Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0674369Z Ran 1 test in 0.002s
2022-12-01T10:31:02.0674374Z 
2022-12-01T10:31:02.0674444Z OK (skipped=1)
2022-12-01T10:31:02.0674449Z 
2022-12-01T10:31:02.0674536Z Generating XML reports...
2022-12-01T10:31:02.0674862Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102615.xml
2022-12-01T10:31:02.0675093Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.0675367Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0675490Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0675770Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0675908Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0675912Z 
2022-12-01T10:31:02.0675985Z Running tests...
2022-12-01T10:31:02.0676179Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0676505Z   test_ddp_profiling_torch_profiler (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'nccl', 'gloo'} (0.002s)
2022-12-01T10:31:02.0676512Z 
2022-12-01T10:31:02.0676701Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0676764Z Ran 1 test in 0.002s
2022-12-01T10:31:02.0676768Z 
2022-12-01T10:31:02.0676839Z OK (skipped=1)
2022-12-01T10:31:02.0676843Z 
2022-12-01T10:31:02.0676927Z Generating XML reports...
2022-12-01T10:31:02.0677264Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102616.xml
2022-12-01T10:31:02.0677499Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.0677773Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0677898Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0678184Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0678308Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0678326Z 
2022-12-01T10:31:02.0678386Z Running tests...
2022-12-01T10:31:02.0678581Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0678818Z   test_ddp_python_error_logged (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24476
2022-12-01T10:31:02.0678973Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24477
2022-12-01T10:31:02.0679126Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 24478
2022-12-01T10:31:02.0679400Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0679523Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0679830Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0679978Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0680159Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:31:02.0680429Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0680552Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0680829Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0680965Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0681142Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:31:02.0681415Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0681526Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0681804Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0681938Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0682115Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:31:02.0682414Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0682712Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0683005Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0683174Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:31:02.0683340Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:31:02.0683490Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:31:02.0683594Z skip: Need at least 2 CUDA devices (1.855s)
2022-12-01T10:31:02.0683599Z 
2022-12-01T10:31:02.0683794Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0683870Z Ran 1 test in 1.855s
2022-12-01T10:31:02.0683874Z 
2022-12-01T10:31:02.0683946Z OK (skipped=1)
2022-12-01T10:31:02.0683950Z 
2022-12-01T10:31:02.0684035Z Generating XML reports...
2022-12-01T10:31:02.0684374Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102617.xml
2022-12-01T10:31:02.0684608Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.0684882Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0684992Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0685272Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0685407Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0685412Z 
2022-12-01T10:31:02.0685485Z Running tests...
2022-12-01T10:31:02.0685680Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0686410Z   test_ddp_returns_tensor_with_no_grad (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78595 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.237s)
2022-12-01T10:31:02.0686439Z 
2022-12-01T10:31:02.0686631Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0686706Z Ran 1 test in 0.237s
2022-12-01T10:31:02.0686711Z 
2022-12-01T10:31:02.0686784Z OK (skipped=1)
2022-12-01T10:31:02.0686788Z 
2022-12-01T10:31:02.0686873Z Generating XML reports...
2022-12-01T10:31:02.0687197Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102621.xml
2022-12-01T10:31:02.0687429Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.0687704Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0687829Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0688112Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0688248Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0688254Z 
2022-12-01T10:31:02.0688327Z Running tests...
2022-12-01T10:31:02.0688522Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0688844Z   test_ddp_shared_grad_acc_unused_params (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'nccl', 'gloo'} (0.002s)
2022-12-01T10:31:02.0688863Z 
2022-12-01T10:31:02.0689041Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0689115Z Ran 1 test in 0.002s
2022-12-01T10:31:02.0689120Z 
2022-12-01T10:31:02.0689192Z OK (skipped=1)
2022-12-01T10:31:02.0689196Z 
2022-12-01T10:31:02.0689280Z Generating XML reports...
2022-12-01T10:31:02.0689619Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102622.xml
2022-12-01T10:31:02.0689851Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.0690128Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0690251Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0690515Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0690650Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0690654Z 
2022-12-01T10:31:02.0690728Z Running tests...
2022-12-01T10:31:02.0690921Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0691621Z   test_ddp_static_graph_nested_types (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77625 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.240s)
2022-12-01T10:31:02.0691629Z 
2022-12-01T10:31:02.0691823Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0691900Z Ran 1 test in 0.241s
2022-12-01T10:31:02.0691905Z 
2022-12-01T10:31:02.0691978Z OK (skipped=1)
2022-12-01T10:31:02.0691983Z 
2022-12-01T10:31:02.0692067Z Generating XML reports...
2022-12-01T10:31:02.0692405Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102624.xml
2022-12-01T10:31:02.0692623Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.0692898Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0693049Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0693330Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0693505Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0693510Z 
2022-12-01T10:31:02.0693583Z Running tests...
2022-12-01T10:31:02.0693777Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0694016Z   test_ddp_sync_bn_training_vs_eval (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24559
2022-12-01T10:31:02.0694172Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24560
2022-12-01T10:31:02.0694313Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 24561
2022-12-01T10:31:02.0694584Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0694709Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0694990Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0695127Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0695388Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:31:02.0695664Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0695789Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0696052Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0696187Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0696366Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:31:02.0696636Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0696762Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0697038Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0697173Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0697350Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:31:02.0697648Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0697931Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0698224Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0698401Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:31:02.0698565Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:31:02.0698728Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:31:02.0698833Z skip: Need at least 2 CUDA devices (1.754s)
2022-12-01T10:31:02.0698839Z 
2022-12-01T10:31:02.0699034Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0699110Z Ran 1 test in 1.755s
2022-12-01T10:31:02.0699115Z 
2022-12-01T10:31:02.0699185Z OK (skipped=1)
2022-12-01T10:31:02.0699190Z 
2022-12-01T10:31:02.0699261Z Generating XML reports...
2022-12-01T10:31:02.0699598Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102625.xml
2022-12-01T10:31:02.0699860Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.0700161Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0700284Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0700728Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0700865Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0700871Z 
2022-12-01T10:31:02.0700944Z Running tests...
2022-12-01T10:31:02.0701140Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0701364Z   test_ddp_sync_module_states (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24612
2022-12-01T10:31:02.0701524Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24613
2022-12-01T10:31:02.0701680Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 24614
2022-12-01T10:31:02.0701954Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0702080Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0702360Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0702493Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0702672Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:31:02.0702927Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0703050Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0703331Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0703467Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0703644Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:31:02.0703914Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0704038Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0704314Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0704449Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0704612Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:31:02.0704910Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0705209Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0705501Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0705665Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:31:02.0705824Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:31:02.0705986Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:31:02.0706089Z skip: Need at least 2 CUDA devices (1.853s)
2022-12-01T10:31:02.0706095Z 
2022-12-01T10:31:02.0706288Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0706349Z Ran 1 test in 1.853s
2022-12-01T10:31:02.0706403Z 
2022-12-01T10:31:02.0706478Z OK (skipped=1)
2022-12-01T10:31:02.0706515Z 
2022-12-01T10:31:02.0706601Z Generating XML reports...
2022-12-01T10:31:02.0706943Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102629.xml
2022-12-01T10:31:02.0707173Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.0707446Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0707570Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0707848Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0707969Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0707988Z 
2022-12-01T10:31:02.0708047Z Running tests...
2022-12-01T10:31:02.0708243Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0708493Z   test_ddp_uneven_input_exception (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24665
2022-12-01T10:31:02.0708652Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24666
2022-12-01T10:31:02.0708808Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 24667
2022-12-01T10:31:02.0709083Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0709207Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0709487Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0709609Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0709789Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:31:02.0710066Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0710189Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0710467Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0710601Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0710779Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:31:02.0711050Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0711173Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0711438Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0711573Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0711753Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:31:02.0712055Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0712352Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0712646Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0712811Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:31:02.0712976Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:31:02.0713168Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:31:02.0713284Z skip: Need at least 2 CUDA devices (1.853s)
2022-12-01T10:31:02.0713289Z 
2022-12-01T10:31:02.0713523Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0713600Z Ran 1 test in 1.854s
2022-12-01T10:31:02.0713605Z 
2022-12-01T10:31:02.0713663Z OK (skipped=1)
2022-12-01T10:31:02.0713682Z 
2022-12-01T10:31:02.0713753Z Generating XML reports...
2022-12-01T10:31:02.0714095Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102632.xml
2022-12-01T10:31:02.0714325Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.0714599Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0714723Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0715006Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0715145Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0715150Z 
2022-12-01T10:31:02.0715223Z Running tests...
2022-12-01T10:31:02.0715403Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0716102Z   test_ddp_uneven_input_join_disable (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78684 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.245s)
2022-12-01T10:31:02.0716107Z 
2022-12-01T10:31:02.0716303Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0716378Z Ran 1 test in 0.245s
2022-12-01T10:31:02.0716383Z 
2022-12-01T10:31:02.0716457Z OK (skipped=1)
2022-12-01T10:31:02.0716461Z 
2022-12-01T10:31:02.0716549Z Generating XML reports...
2022-12-01T10:31:02.0716889Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102635.xml
2022-12-01T10:31:02.0717121Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.0717396Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0717521Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0717788Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0717923Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0717928Z 
2022-12-01T10:31:02.0718001Z Running tests...
2022-12-01T10:31:02.0718200Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0718878Z   test_ddp_uneven_inputs (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/75648 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.238s)
2022-12-01T10:31:02.0718887Z 
2022-12-01T10:31:02.0719079Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0719155Z Ran 1 test in 0.238s
2022-12-01T10:31:02.0719160Z 
2022-12-01T10:31:02.0719236Z OK (skipped=1)
2022-12-01T10:31:02.0719240Z 
2022-12-01T10:31:02.0719324Z Generating XML reports...
2022-12-01T10:31:02.0719650Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102637.xml
2022-12-01T10:31:02.0719911Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.0720188Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0720336Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0720617Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0720752Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0720757Z 
2022-12-01T10:31:02.0720831Z Running tests...
2022-12-01T10:31:02.0721028Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0721741Z   test_ddp_uneven_inputs_stop_iteration_sync_bn (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78113 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.237s)
2022-12-01T10:31:02.0721750Z 
2022-12-01T10:31:02.0721942Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0722004Z Ran 1 test in 0.237s
2022-12-01T10:31:02.0722020Z 
2022-12-01T10:31:02.0722078Z OK (skipped=1)
2022-12-01T10:31:02.0722082Z 
2022-12-01T10:31:02.0722165Z Generating XML reports...
2022-12-01T10:31:02.0722500Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102639.xml
2022-12-01T10:31:02.0722731Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.0723003Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0723125Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0723407Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0723544Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0723548Z 
2022-12-01T10:31:02.0723611Z Running tests...
2022-12-01T10:31:02.0723806Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0724155Z   test_ddp_unused_params_rebuild_buckets_exception (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'nccl', 'gloo'} (0.003s)
2022-12-01T10:31:02.0724161Z 
2022-12-01T10:31:02.0724350Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0724426Z Ran 1 test in 0.003s
2022-12-01T10:31:02.0724430Z 
2022-12-01T10:31:02.0724500Z OK (skipped=1)
2022-12-01T10:31:02.0724505Z 
2022-12-01T10:31:02.0724588Z Generating XML reports...
2022-12-01T10:31:02.0724925Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102640.xml
2022-12-01T10:31:02.0725158Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.0725418Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0725542Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0725822Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0725958Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0725963Z 
2022-12-01T10:31:02.0726035Z Running tests...
2022-12-01T10:31:02.0726230Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0726462Z   test_destroy_full_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24758
2022-12-01T10:31:02.0726649Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24759
2022-12-01T10:31:02.0726830Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 24760
2022-12-01T10:31:02.0727090Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0727212Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0727490Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0727625Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0727804Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:31:02.0728074Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0728196Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0728476Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0728601Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0728778Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:31:02.0729076Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0729342Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0729464Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0729744Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0729877Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0730054Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:31:02.0730348Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0730630Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0730795Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:31:02.0730957Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:31:02.0731118Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:31:02.0731293Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T10:31:02.0731463Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T10:31:02.0731636Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-12-01T10:31:02.0731933Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:31:02.0732243Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:31:02.0732517Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:31:02.0732585Z ok (1.866s)
2022-12-01T10:31:02.0732590Z 
2022-12-01T10:31:02.0732784Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0732860Z Ran 1 test in 1.866s
2022-12-01T10:31:02.0732865Z 
2022-12-01T10:31:02.0732925Z OK
2022-12-01T10:31:02.0732930Z 
2022-12-01T10:31:02.0733012Z Generating XML reports...
2022-12-01T10:31:02.0733380Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102642.xml
2022-12-01T10:31:02.0733635Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.0733907Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0734019Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0734298Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0734431Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0734436Z 
2022-12-01T10:31:02.0734510Z Running tests...
2022-12-01T10:31:02.0734703Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0734933Z   test_destroy_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24820
2022-12-01T10:31:02.0735088Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24821
2022-12-01T10:31:02.0735310Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 24822
2022-12-01T10:31:02.0735579Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0735703Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0735982Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0736118Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0736296Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:31:02.0736566Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0736691Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0736972Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0737109Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0737272Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:31:02.0737540Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0737664Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0737940Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0738076Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0738255Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:31:02.0738549Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0738847Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0739136Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0739290Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:31:02.0739453Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:31:02.0739615Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:31:02.0739789Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T10:31:02.0739999Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T10:31:02.0740223Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-12-01T10:31:02.0740608Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:31:02.0740897Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:31:02.0741186Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:31:02.0741242Z ok (2.053s)
2022-12-01T10:31:02.0741248Z 
2022-12-01T10:31:02.0741446Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0741521Z Ran 1 test in 2.053s
2022-12-01T10:31:02.0741527Z 
2022-12-01T10:31:02.0741590Z OK
2022-12-01T10:31:02.0741595Z 
2022-12-01T10:31:02.0741680Z Generating XML reports...
2022-12-01T10:31:02.0742023Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102645.xml
2022-12-01T10:31:02.0742258Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.0742531Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0742641Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0742924Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0743059Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0743064Z 
2022-12-01T10:31:02.0743138Z Running tests...
2022-12-01T10:31:02.0743331Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0744077Z   test_detect_ddp_is_actually_static (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78767 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.238s)
2022-12-01T10:31:02.0744085Z 
2022-12-01T10:31:02.0744277Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0744350Z Ran 1 test in 0.239s
2022-12-01T10:31:02.0744355Z 
2022-12-01T10:31:02.0744425Z OK (skipped=1)
2022-12-01T10:31:02.0744429Z 
2022-12-01T10:31:02.0744512Z Generating XML reports...
2022-12-01T10:31:02.0744841Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102648.xml
2022-12-01T10:31:02.0745071Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.0745346Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0745470Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0745749Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0745884Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0745889Z 
2022-12-01T10:31:02.0745962Z Running tests...
2022-12-01T10:31:02.0746157Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0746478Z   test_different_graph_across_ranks (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'nccl', 'gloo'} (0.002s)
2022-12-01T10:31:02.0746484Z 
2022-12-01T10:31:02.0746662Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0746737Z Ran 1 test in 0.002s
2022-12-01T10:31:02.0746741Z 
2022-12-01T10:31:02.0746882Z OK (skipped=1)
2022-12-01T10:31:02.0746888Z 
2022-12-01T10:31:02.0747006Z Generating XML reports...
2022-12-01T10:31:02.0747346Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102650.xml
2022-12-01T10:31:02.0747579Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.0747855Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0747978Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0748258Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0748382Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0748387Z 
2022-12-01T10:31:02.0748461Z Running tests...
2022-12-01T10:31:02.0748657Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0748900Z   test_dump_DDP_relevant_env_vars (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24899
2022-12-01T10:31:02.0749061Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24900
2022-12-01T10:31:02.0749215Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 24901
2022-12-01T10:31:02.0749545Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0749717Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0749987Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0750121Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0750299Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:31:02.0750571Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0750695Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0750972Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0751106Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0751280Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:31:02.0751547Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0751656Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0751931Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0752068Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0752245Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:31:02.0752540Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0752834Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0753123Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0753288Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:31:02.0753450Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:31:02.0753640Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:31:02.0753708Z ok (1.752s)
2022-12-01T10:31:02.0753741Z 
2022-12-01T10:31:02.0753936Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0754010Z Ran 1 test in 1.753s
2022-12-01T10:31:02.0754015Z 
2022-12-01T10:31:02.0754076Z OK
2022-12-01T10:31:02.0754080Z 
2022-12-01T10:31:02.0754163Z Generating XML reports...
2022-12-01T10:31:02.0754504Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102651.xml
2022-12-01T10:31:02.0754736Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.0754999Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0755122Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0755401Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0755537Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0755544Z 
2022-12-01T10:31:02.0755616Z Running tests...
2022-12-01T10:31:02.0755810Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0756030Z   test_gather (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24952
2022-12-01T10:31:02.0756188Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24953
2022-12-01T10:31:02.0756340Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 24954
2022-12-01T10:31:02.0756598Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0756721Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0757000Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0757136Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0757314Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:31:02.0757580Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0757702Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0757978Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0758111Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0758275Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:31:02.0758542Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0758665Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0758944Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0759076Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0759253Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:31:02.0759551Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0759845Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0760132Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0760313Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:31:02.0760473Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:31:02.0760750Z STAGE:2022-12-01 10:26:56 24953:24953 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0760991Z STAGE:2022-12-01 10:26:56 24954:24954 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0761156Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:31:02.0761396Z STAGE:2022-12-01 10:26:56 24952:24952 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0761642Z STAGE:2022-12-01 10:26:56 24953:24953 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.0761885Z STAGE:2022-12-01 10:26:56 24954:24954 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.0762113Z STAGE:2022-12-01 10:26:56 24952:24952 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.0762352Z STAGE:2022-12-01 10:26:56 24953:24953 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0762589Z STAGE:2022-12-01 10:26:56 24954:24954 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0762836Z STAGE:2022-12-01 10:26:56 24954:24954 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.0763075Z STAGE:2022-12-01 10:26:56 24954:24954 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0763310Z STAGE:2022-12-01 10:26:56 24952:24952 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0763553Z STAGE:2022-12-01 10:26:56 24952:24952 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.0763794Z STAGE:2022-12-01 10:26:56 24953:24953 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.0764037Z STAGE:2022-12-01 10:26:56 24952:24952 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0764269Z STAGE:2022-12-01 10:26:56 24952:24952 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.0764512Z STAGE:2022-12-01 10:26:56 24953:24953 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0764754Z STAGE:2022-12-01 10:26:56 24953:24953 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.0764994Z STAGE:2022-12-01 10:26:56 24954:24954 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.0765062Z ok (1.954s)
2022-12-01T10:31:02.0765067Z 
2022-12-01T10:31:02.0765263Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0765338Z Ran 1 test in 1.954s
2022-12-01T10:31:02.0765343Z 
2022-12-01T10:31:02.0765403Z OK
2022-12-01T10:31:02.0765407Z 
2022-12-01T10:31:02.0765494Z Generating XML reports...
2022-12-01T10:31:02.0765824Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102655.xml
2022-12-01T10:31:02.0766057Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.0766330Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0766454Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0766735Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0766872Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0766877Z 
2022-12-01T10:31:02.0766949Z Running tests...
2022-12-01T10:31:02.0767140Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0767355Z   test_gather_checks (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25008
2022-12-01T10:31:02.0767544Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25009
2022-12-01T10:31:02.0767724Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 25010
2022-12-01T10:31:02.0767999Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0768121Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0768397Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0768533Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0768713Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:31:02.0768984Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0769096Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0769373Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0769509Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0769685Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:31:02.0769986Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0770255Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0770380Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0770658Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0770822Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0770983Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:31:02.0771277Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0771574Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0771744Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:31:02.0771908Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:31:02.0772072Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:31:02.0772140Z ok (1.755s)
2022-12-01T10:31:02.0772145Z 
2022-12-01T10:31:02.0772341Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0772405Z Ran 1 test in 1.756s
2022-12-01T10:31:02.0772424Z 
2022-12-01T10:31:02.0772475Z OK
2022-12-01T10:31:02.0772480Z 
2022-12-01T10:31:02.0772565Z Generating XML reports...
2022-12-01T10:31:02.0772906Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102658.xml
2022-12-01T10:31:02.0773137Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.0773413Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0773538Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0773819Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0773956Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0773961Z 
2022-12-01T10:31:02.0774021Z Running tests...
2022-12-01T10:31:02.0774244Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0774457Z   test_gather_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA gather (0.002s)
2022-12-01T10:31:02.0774462Z 
2022-12-01T10:31:02.0774655Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0774731Z Ran 1 test in 0.002s
2022-12-01T10:31:02.0774736Z 
2022-12-01T10:31:02.0774809Z OK (skipped=1)
2022-12-01T10:31:02.0774814Z 
2022-12-01T10:31:02.0774899Z Generating XML reports...
2022-12-01T10:31:02.0775302Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102701.xml
2022-12-01T10:31:02.0775538Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.0775798Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0775925Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0776206Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0776344Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0776349Z 
2022-12-01T10:31:02.0776422Z Running tests...
2022-12-01T10:31:02.0776617Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0776847Z   test_gather_full_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25071
2022-12-01T10:31:02.0777005Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25072
2022-12-01T10:31:02.0777148Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 25073
2022-12-01T10:31:02.0777421Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0777546Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0777827Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0777960Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0778137Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:31:02.0778407Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0778530Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0778808Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0778930Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0779108Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:31:02.0779377Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0779501Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0779776Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0779907Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0780079Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:31:02.0780474Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0780774Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0781114Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0781390Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:31:02.0781554Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:31:02.0781716Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:31:02.0781892Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T10:31:02.0782067Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T10:31:02.0782240Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-12-01T10:31:02.0782537Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:31:02.0808546Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:31:02.0808967Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:31:02.0809384Z STAGE:2022-12-01 10:27:04 25072:25072 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-12-01 10:27:04 25071:25071 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0809392Z 
2022-12-01T10:31:02.0809629Z STAGE:2022-12-01 10:27:04 25073:25073 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0809871Z STAGE:2022-12-01 10:27:04 25072:25072 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.0810108Z STAGE:2022-12-01 10:27:04 25073:25073 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.0810344Z STAGE:2022-12-01 10:27:04 25071:25071 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.0810572Z STAGE:2022-12-01 10:27:04 25073:25073 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0810803Z STAGE:2022-12-01 10:27:04 25072:25072 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0811037Z STAGE:2022-12-01 10:27:04 25073:25073 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.0811264Z STAGE:2022-12-01 10:27:04 25073:25073 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0811489Z STAGE:2022-12-01 10:27:04 25071:25071 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0811721Z STAGE:2022-12-01 10:27:04 25071:25071 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.0811949Z STAGE:2022-12-01 10:27:04 25072:25072 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.0812177Z STAGE:2022-12-01 10:27:04 25071:25071 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0812407Z STAGE:2022-12-01 10:27:04 25071:25071 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.0812638Z STAGE:2022-12-01 10:27:04 25072:25072 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0812874Z STAGE:2022-12-01 10:27:04 25072:25072 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.0813109Z STAGE:2022-12-01 10:27:04 25073:25073 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.0813169Z ok (2.053s)
2022-12-01T10:31:02.0813174Z 
2022-12-01T10:31:02.0813364Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0813432Z Ran 1 test in 2.053s
2022-12-01T10:31:02.0813437Z 
2022-12-01T10:31:02.0813489Z OK
2022-12-01T10:31:02.0813494Z 
2022-12-01T10:31:02.0813569Z Generating XML reports...
2022-12-01T10:31:02.0814000Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102702.xml
2022-12-01T10:31:02.0814281Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.0814551Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0814668Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0814942Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0815069Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0815074Z 
2022-12-01T10:31:02.0815140Z Running tests...
2022-12-01T10:31:02.0815404Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0815625Z   test_gather_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25136
2022-12-01T10:31:02.0815774Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25137
2022-12-01T10:31:02.0815924Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 25138
2022-12-01T10:31:02.0816191Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0816307Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0816580Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0816704Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0816877Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:31:02.0817142Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0817254Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0817528Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0817656Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0817827Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:31:02.0818090Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0818206Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0818479Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0818607Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0818779Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:31:02.0819065Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0819355Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0819642Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0819802Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:31:02.0819960Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:31:02.0820127Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-12-01T10:31:02.0820292Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T10:31:02.0820743Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:31:02.0820940Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T10:31:02.0821225Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:31:02.0821509Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:31:02.0821792Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:31:02.0822034Z STAGE:2022-12-01 10:27:08 25137:25137 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0822266Z STAGE:2022-12-01 10:27:08 25138:25138 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0822506Z STAGE:2022-12-01 10:27:08 25138:25138 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.0822744Z STAGE:2022-12-01 10:27:08 25137:25137 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.0822974Z STAGE:2022-12-01 10:27:08 25138:25138 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0823201Z STAGE:2022-12-01 10:27:08 25137:25137 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0823429Z STAGE:2022-12-01 10:27:08 25137:25137 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.0823659Z STAGE:2022-12-01 10:27:08 25138:25138 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.0823719Z ok (2.051s)
2022-12-01T10:31:02.0823724Z 
2022-12-01T10:31:02.0823913Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0823981Z Ran 1 test in 2.051s
2022-12-01T10:31:02.0823987Z 
2022-12-01T10:31:02.0824039Z OK
2022-12-01T10:31:02.0824044Z 
2022-12-01T10:31:02.0824120Z Generating XML reports...
2022-12-01T10:31:02.0824461Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102706.xml
2022-12-01T10:31:02.0824688Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.0824950Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0825066Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0825339Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0825466Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0825471Z 
2022-12-01T10:31:02.0825536Z Running tests...
2022-12-01T10:31:02.0825723Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0825946Z   test_gather_object (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25197
2022-12-01T10:31:02.0826098Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25198
2022-12-01T10:31:02.0826240Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 25199
2022-12-01T10:31:02.0826504Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0826619Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0826893Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0827017Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0827185Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:31:02.0827488Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0827605Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0827904Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0828026Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0828195Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:31:02.0828486Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0828750Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0828864Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0829135Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0829262Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0829429Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:31:02.0829717Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0830004Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0830159Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:31:02.0830313Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:31:02.0830467Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:31:02.0830526Z ok (1.752s)
2022-12-01T10:31:02.0830531Z 
2022-12-01T10:31:02.0830719Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0830789Z Ran 1 test in 1.752s
2022-12-01T10:31:02.0830794Z 
2022-12-01T10:31:02.0830848Z OK
2022-12-01T10:31:02.0830853Z 
2022-12-01T10:31:02.0830924Z Generating XML reports...
2022-12-01T10:31:02.0831255Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102709.xml
2022-12-01T10:31:02.0831479Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.0831742Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0831858Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0832131Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0832258Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0832265Z 
2022-12-01T10:31:02.0832329Z Running tests...
2022-12-01T10:31:02.0832520Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0833198Z   test_gather_object_subgroup (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/82866 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.235s)
2022-12-01T10:31:02.0833210Z 
2022-12-01T10:31:02.0833393Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0833462Z Ran 1 test in 0.235s
2022-12-01T10:31:02.0833467Z 
2022-12-01T10:31:02.0833532Z OK (skipped=1)
2022-12-01T10:31:02.0833536Z 
2022-12-01T10:31:02.0833614Z Generating XML reports...
2022-12-01T10:31:02.0833975Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102713.xml
2022-12-01T10:31:02.0834231Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.0834496Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0834611Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0834880Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0835003Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0835008Z 
2022-12-01T10:31:02.0835072Z Running tests...
2022-12-01T10:31:02.0835260Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0835477Z   test_get_backend (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25260
2022-12-01T10:31:02.0835627Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25261
2022-12-01T10:31:02.0835778Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 25262
2022-12-01T10:31:02.0836043Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0836159Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0836425Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0836551Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0836719Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:31:02.0836986Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0837104Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0837377Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0837506Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0837676Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:31:02.0837936Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0838045Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0838313Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0838438Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0838605Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:31:02.0838899Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0839195Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0839479Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0839637Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:31:02.0839792Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:31:02.0839936Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:31:02.0840104Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T10:31:02.0840296Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T10:31:02.0840488Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-12-01T10:31:02.0840775Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:31:02.0841064Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:31:02.0841346Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:31:02.0841407Z ok (1.854s)
2022-12-01T10:31:02.0841412Z 
2022-12-01T10:31:02.0841601Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0841664Z Ran 1 test in 1.854s
2022-12-01T10:31:02.0841669Z 
2022-12-01T10:31:02.0841722Z OK
2022-12-01T10:31:02.0841726Z 
2022-12-01T10:31:02.0841802Z Generating XML reports...
2022-12-01T10:31:02.0842139Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102714.xml
2022-12-01T10:31:02.0842369Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.0842636Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0842751Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0843024Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0843148Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0843158Z 
2022-12-01T10:31:02.0843218Z Running tests...
2022-12-01T10:31:02.0843405Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0843623Z   test_get_future (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25319
2022-12-01T10:31:02.0843774Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25320
2022-12-01T10:31:02.0843921Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 25321
2022-12-01T10:31:02.0844189Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0844306Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0844581Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0844701Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0844879Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:31:02.0845158Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0845285Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0845567Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0845704Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0845882Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:31:02.0846152Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0846265Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0846544Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0846679Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0846883Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:31:02.0847214Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0847511Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0847804Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0847970Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:31:02.0848139Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:31:02.0848287Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:31:02.0848356Z ok (1.753s)
2022-12-01T10:31:02.0848361Z 
2022-12-01T10:31:02.0848557Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0848637Z Ran 1 test in 1.753s
2022-12-01T10:31:02.0848642Z 
2022-12-01T10:31:02.0848704Z OK
2022-12-01T10:31:02.0848709Z 
2022-12-01T10:31:02.0848793Z Generating XML reports...
2022-12-01T10:31:02.0849136Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102718.xml
2022-12-01T10:31:02.0849369Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.0849644Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0849753Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0850033Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0850168Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0850175Z 
2022-12-01T10:31:02.0850250Z Running tests...
2022-12-01T10:31:02.0850447Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0850670Z   test_get_rank (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25372
2022-12-01T10:31:02.0850829Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25373
2022-12-01T10:31:02.0850985Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 25374
2022-12-01T10:31:02.0851246Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0851370Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0851653Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0851792Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0851971Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:31:02.0852244Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0852368Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0852647Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0852783Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0852946Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:31:02.0853219Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0853343Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0853652Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0853816Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0853995Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:31:02.0854294Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0854591Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0854884Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0855037Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:31:02.0855206Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:31:02.0855467Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:31:02.0855543Z ok (2.054s)
2022-12-01T10:31:02.0855548Z 
2022-12-01T10:31:02.0855749Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0855826Z Ran 1 test in 2.054s
2022-12-01T10:31:02.0855831Z 
2022-12-01T10:31:02.0855894Z OK
2022-12-01T10:31:02.0855899Z 
2022-12-01T10:31:02.0855984Z Generating XML reports...
2022-12-01T10:31:02.0856314Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102721.xml
2022-12-01T10:31:02.0856546Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.0856820Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0856944Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0857226Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0857363Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0857368Z 
2022-12-01T10:31:02.0857443Z Running tests...
2022-12-01T10:31:02.0857638Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0857878Z   test_get_rank_size_full_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25425
2022-12-01T10:31:02.0858024Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25426
2022-12-01T10:31:02.0858178Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 25427
2022-12-01T10:31:02.0858450Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0858576Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0858858Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0858996Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0859173Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:31:02.0859444Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0859568Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0859832Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0859967Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0860178Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:31:02.0860567Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0860752Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0861037Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0861172Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0861348Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:31:02.0861635Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0861931Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0862228Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0862398Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:31:02.0862563Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:31:02.0862723Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:31:02.0862900Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T10:31:02.0863073Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-12-01T10:31:02.0863248Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T10:31:02.0863528Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:31:02.0863822Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:31:02.0864118Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:31:02.0864188Z ok (2.053s)
2022-12-01T10:31:02.0864194Z 
2022-12-01T10:31:02.0864391Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0864467Z Ran 1 test in 2.053s
2022-12-01T10:31:02.0864472Z 
2022-12-01T10:31:02.0864534Z OK
2022-12-01T10:31:02.0864538Z 
2022-12-01T10:31:02.0864624Z Generating XML reports...
2022-12-01T10:31:02.0864962Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102724.xml
2022-12-01T10:31:02.0865181Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.0865458Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0865582Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0865864Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0866000Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0866005Z 
2022-12-01T10:31:02.0866079Z Running tests...
2022-12-01T10:31:02.0866276Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0866511Z   test_get_rank_size_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25487
2022-12-01T10:31:02.0866670Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25488
2022-12-01T10:31:02.0866812Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 25489
2022-12-01T10:31:02.0867122Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0867275Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0867556Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0867693Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0867870Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:31:02.0868141Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0868266Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0868530Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0868664Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0868842Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:31:02.0869114Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0869237Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0869519Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0869655Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0869832Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:31:02.0870131Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0870416Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0870712Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0870882Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:31:02.0871045Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:31:02.0871205Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:31:02.0871388Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T10:31:02.0871562Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T10:31:02.0871735Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-12-01T10:31:02.0872033Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:31:02.0872312Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:31:02.0872603Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:31:02.0872672Z ok (1.958s)
2022-12-01T10:31:02.0872677Z 
2022-12-01T10:31:02.0872878Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0872958Z Ran 1 test in 1.959s
2022-12-01T10:31:02.0872964Z 
2022-12-01T10:31:02.0873027Z OK
2022-12-01T10:31:02.0873032Z 
2022-12-01T10:31:02.0873118Z Generating XML reports...
2022-12-01T10:31:02.0873463Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102728.xml
2022-12-01T10:31:02.0873697Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.0873989Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0874166Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0874449Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0874592Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0874597Z 
2022-12-01T10:31:02.0874670Z Running tests...
2022-12-01T10:31:02.0874866Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0875179Z   test_invalid_static_graph (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'nccl', 'gloo'} (0.002s)
2022-12-01T10:31:02.0875185Z 
2022-12-01T10:31:02.0875380Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0875456Z Ran 1 test in 0.003s
2022-12-01T10:31:02.0875461Z 
2022-12-01T10:31:02.0875523Z OK (skipped=1)
2022-12-01T10:31:02.0875528Z 
2022-12-01T10:31:02.0875616Z Generating XML reports...
2022-12-01T10:31:02.0875954Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102731.xml
2022-12-01T10:31:02.0876186Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.0876461Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0876585Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0876863Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0876999Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0877003Z 
2022-12-01T10:31:02.0877063Z Running tests...
2022-12-01T10:31:02.0877263Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0877484Z   test_irecv (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25556
2022-12-01T10:31:02.0877647Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25557
2022-12-01T10:31:02.0877803Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 25558
2022-12-01T10:31:02.0878075Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0878199Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0878480Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0878615Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0878780Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:31:02.0879054Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0879183Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0879461Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0879597Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0879775Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:31:02.0880049Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0880176Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0880440Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0880605Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0880812Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:31:02.0881113Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0881414Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0881707Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0881873Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:31:02.0882038Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:31:02.0882200Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:31:02.0882259Z ok (1.955s)
2022-12-01T10:31:02.0882283Z 
2022-12-01T10:31:02.0882469Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0882546Z Ran 1 test in 1.955s
2022-12-01T10:31:02.0882552Z 
2022-12-01T10:31:02.0882617Z OK
2022-12-01T10:31:02.0882621Z 
2022-12-01T10:31:02.0882707Z Generating XML reports...
2022-12-01T10:31:02.0883048Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102733.xml
2022-12-01T10:31:02.0883281Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.0883555Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0883681Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0883948Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0884086Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0884093Z 
2022-12-01T10:31:02.0884168Z Running tests...
2022-12-01T10:31:02.0884366Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0884584Z   test_isend (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25609
2022-12-01T10:31:02.0884742Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25610
2022-12-01T10:31:02.0884897Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 25611
2022-12-01T10:31:02.0885169Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0885279Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0885560Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0885696Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0885874Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:31:02.0886144Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0886268Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0886549Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0886685Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0886863Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:31:02.0887120Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0887270Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0887576Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0887710Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0887887Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:31:02.0888186Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0888485Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0888779Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0888946Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:31:02.0889096Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:31:02.0889262Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:31:02.0889331Z ok (1.856s)
2022-12-01T10:31:02.0889336Z 
2022-12-01T10:31:02.0889533Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0889609Z Ran 1 test in 1.856s
2022-12-01T10:31:02.0889614Z 
2022-12-01T10:31:02.0889678Z OK
2022-12-01T10:31:02.0889683Z 
2022-12-01T10:31:02.0889768Z Generating XML reports...
2022-12-01T10:31:02.0890112Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102736.xml
2022-12-01T10:31:02.0890334Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.0890614Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0890741Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0891024Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0891162Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0891167Z 
2022-12-01T10:31:02.0891241Z Running tests...
2022-12-01T10:31:02.0891438Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0891678Z   test_isend_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25662
2022-12-01T10:31:02.0891837Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25663
2022-12-01T10:31:02.0891982Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 25664
2022-12-01T10:31:02.0892257Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0892384Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0892663Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0892800Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0892976Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:31:02.0893250Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0893374Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0893641Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0893779Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0893990Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:31:02.0894318Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0894589Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0894715Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0894993Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0895131Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0895393Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:31:02.0895767Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0896181Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0896352Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:31:02.0896581Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:31:02.0896966Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:31:02.0897414Z STAGE:2022-12-01 10:27:41 25662:25662 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0897900Z STAGE:2022-12-01 10:27:41 25663:25663 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0898337Z STAGE:2022-12-01 10:27:41 25664:25664 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0898784Z STAGE:2022-12-01 10:27:41 25662:25662 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.0899219Z STAGE:2022-12-01 10:27:41 25663:25663 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.0899636Z STAGE:2022-12-01 10:27:41 25664:25664 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.0899899Z ok (1.851s)
2022-12-01T10:31:02.0900004Z 
2022-12-01T10:31:02.0900206Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0900532Z Ran 1 test in 1.851s
2022-12-01T10:31:02.0900651Z 
2022-12-01T10:31:02.0900714Z OK
2022-12-01T10:31:02.0900809Z 
2022-12-01T10:31:02.0900896Z Generating XML reports...
2022-12-01T10:31:02.0901367Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102739.xml
2022-12-01T10:31:02.0901854Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.0902328Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0902668Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0903092Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0903446Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0903614Z 
2022-12-01T10:31:02.0903690Z Running tests...
2022-12-01T10:31:02.0903996Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0904391Z   test_isend_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25718
2022-12-01T10:31:02.0904799Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25719
2022-12-01T10:31:02.0905136Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 25720
2022-12-01T10:31:02.0905720Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0906097Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0906535Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0906888Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0907217Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:31:02.0907697Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0908039Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0908472Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0908825Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0909173Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:31:02.0909668Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0910000Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0910413Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0910759Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0911093Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:31:02.0911591Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0912108Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0912624Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0913016Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:31:02.0913455Z STAGE:2022-12-01 10:27:44 25719:25719 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0913797Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:31:02.0914144Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:31:02.0914576Z STAGE:2022-12-01 10:27:44 25718:25718 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0914992Z STAGE:2022-12-01 10:27:44 25720:25720 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.0915440Z STAGE:2022-12-01 10:27:44 25720:25720 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.0915878Z STAGE:2022-12-01 10:27:44 25719:25719 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.0916305Z STAGE:2022-12-01 10:27:44 25718:25718 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.0916558Z ok (1.954s)
2022-12-01T10:31:02.0916666Z 
2022-12-01T10:31:02.0916867Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0917110Z Ran 1 test in 1.955s
2022-12-01T10:31:02.0917223Z 
2022-12-01T10:31:02.0917285Z OK
2022-12-01T10:31:02.0917364Z 
2022-12-01T10:31:02.0917447Z Generating XML reports...
2022-12-01T10:31:02.0917916Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102743.xml
2022-12-01T10:31:02.0918433Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.0918944Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0919319Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0919751Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0920101Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0920265Z 
2022-12-01T10:31:02.0920325Z Running tests...
2022-12-01T10:31:02.0920629Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0921137Z   test_monitored_barrier_allreduce_hang (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'gloo', 'nccl'} (0.001s)
2022-12-01T10:31:02.0921382Z 
2022-12-01T10:31:02.0921580Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0921816Z Ran 1 test in 0.002s
2022-12-01T10:31:02.0921931Z 
2022-12-01T10:31:02.0922004Z OK (skipped=1)
2022-12-01T10:31:02.0922116Z 
2022-12-01T10:31:02.0922203Z Generating XML reports...
2022-12-01T10:31:02.0922658Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102746.xml
2022-12-01T10:31:02.0923157Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.0923629Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0923975Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0924397Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0924766Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0924932Z 
2022-12-01T10:31:02.0925012Z Running tests...
2022-12-01T10:31:02.0925329Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0925853Z   test_monitored_barrier_allreduce_hang_wait_all_ranks (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'gloo', 'nccl'} (0.001s)
2022-12-01T10:31:02.0926117Z 
2022-12-01T10:31:02.0926312Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0926555Z Ran 1 test in 0.002s
2022-12-01T10:31:02.0926670Z 
2022-12-01T10:31:02.0926744Z OK (skipped=1)
2022-12-01T10:31:02.0926840Z 
2022-12-01T10:31:02.0926929Z Generating XML reports...
2022-12-01T10:31:02.0927388Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102747.xml
2022-12-01T10:31:02.0927887Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.0928355Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0928740Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0929175Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0929526Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0929690Z 
2022-12-01T10:31:02.0929751Z Running tests...
2022-12-01T10:31:02.0930056Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0930474Z   test_monitored_barrier_failure_order (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25794
2022-12-01T10:31:02.0930888Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25795
2022-12-01T10:31:02.0931233Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 25796
2022-12-01T10:31:02.0931708Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0932081Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0932630Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0932991Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0933337Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:31:02.0933824Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0934177Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0934604Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0934953Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0935377Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:31:02.0935907Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0936290Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0937015Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0937526Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0937976Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:31:02.0938498Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0939035Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0939564Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0939948Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:31:02.0940304Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:31:02.0940801Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:31:02.0941254Z [E ProcessGroupGloo.cpp:137] Rank 1 successfully reached monitoredBarrier, but received errors while waiting for send/recv from rank 0. Please check rank 0 logs for faulty rank.
2022-12-01T10:31:02.0941680Z [E ProcessGroupGloo.cpp:137] [Rank 0]: Rank 2 failed to pass monitoredBarrier in 2000 ms
2022-12-01T10:31:02.0941946Z ok (3.955s)
2022-12-01T10:31:02.0942053Z 
2022-12-01T10:31:02.0942264Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0942498Z Ran 1 test in 3.956s
2022-12-01T10:31:02.0942615Z 
2022-12-01T10:31:02.0942680Z OK
2022-12-01T10:31:02.0942777Z 
2022-12-01T10:31:02.0942868Z Generating XML reports...
2022-12-01T10:31:02.0943338Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102749.xml
2022-12-01T10:31:02.0943826Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.0944295Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0944635Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0945070Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0945406Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0945574Z 
2022-12-01T10:31:02.0945651Z Running tests...
2022-12-01T10:31:02.0946062Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0946456Z   test_monitored_barrier_gloo (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25847
2022-12-01T10:31:02.0946989Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25848
2022-12-01T10:31:02.0947324Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 25849
2022-12-01T10:31:02.0947786Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0948112Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0948545Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0948896Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0949230Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:31:02.0949711Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0950044Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0950472Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0950809Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0951181Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:31:02.0951684Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0952187Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0952505Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0952942Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0953294Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0953623Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:31:02.0954123Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0954654Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0955049Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:31:02.0955390Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:31:02.0955742Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:31:02.0956096Z [E ProcessGroupGloo.cpp:137] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 2000 ms
2022-12-01T10:31:02.0956547Z [E ProcessGroupGloo.cpp:137] Rank 2 successfully reached monitoredBarrier, but received errors while waiting for send/recv from rank 0. Please check rank 0 logs for faulty rank.
2022-12-01T10:31:02.0956868Z ok (3.856s)
2022-12-01T10:31:02.0956972Z 
2022-12-01T10:31:02.0957173Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0957422Z Ran 1 test in 3.857s
2022-12-01T10:31:02.0957537Z 
2022-12-01T10:31:02.0957584Z OK
2022-12-01T10:31:02.0957677Z 
2022-12-01T10:31:02.0957763Z Generating XML reports...
2022-12-01T10:31:02.0958229Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102754.xml
2022-12-01T10:31:02.0958728Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.0959230Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0959591Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0960026Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0960368Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0960536Z 
2022-12-01T10:31:02.0960610Z Running tests...
2022-12-01T10:31:02.0960950Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0961398Z   test_monitored_barrier_gloo_rank_0_timeout (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25900
2022-12-01T10:31:02.0961812Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25901
2022-12-01T10:31:02.0962151Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 25902
2022-12-01T10:31:02.0962615Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0962957Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0963382Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0963733Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0964074Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:31:02.0964537Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0964867Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0965297Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0965643Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0965968Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:31:02.0966442Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0966776Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0967192Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0967539Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0967879Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:31:02.0968373Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0968886Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0969404Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0969793Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:31:02.0970144Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:31:02.0970480Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:31:02.0970839Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T10:31:02.0971212Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T10:31:02.0971754Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:31:02.0972141Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-12-01T10:31:02.0972667Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:31:02.0973186Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:31:02.0973551Z [E ProcessGroupGloo.cpp:137] Rank 0 timed out in monitoredBarrier after 0 ms.
2022-12-01T10:31:02.0973855Z No ranks successfully processed in monitoredBarrier.
2022-12-01T10:31:02.0974171Z [E ProcessGroupGloo.cpp:137] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 0 ms
2022-12-01T10:31:02.0974424Z ok (2.054s)
2022-12-01T10:31:02.0974515Z 
2022-12-01T10:31:02.0974714Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0974959Z Ran 1 test in 2.054s
2022-12-01T10:31:02.0975073Z 
2022-12-01T10:31:02.0975137Z OK
2022-12-01T10:31:02.0975313Z 
2022-12-01T10:31:02.0975392Z Generating XML reports...
2022-12-01T10:31:02.0975861Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102800.xml
2022-12-01T10:31:02.0976361Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.0976891Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0977213Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0977648Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0977996Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0978161Z 
2022-12-01T10:31:02.0978221Z Running tests...
2022-12-01T10:31:02.0978532Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0978948Z   test_monitored_barrier_gloo_subgroup (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25962
2022-12-01T10:31:02.0979369Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25963
2022-12-01T10:31:02.0979694Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 25964
2022-12-01T10:31:02.0980150Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0980685Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0981112Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0981461Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0981806Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:31:02.0982287Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0982609Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0983047Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0983399Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0983741Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:31:02.0984201Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0984581Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0985014Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0985423Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0985800Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:31:02.0986307Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0986830Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0987341Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.0987737Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:31:02.0988087Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:31:02.0988439Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:31:02.0988787Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-12-01T10:31:02.0989162Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T10:31:02.0989531Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T10:31:02.0990045Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:31:02.0990566Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:31:02.0991085Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:31:02.0991477Z [E ProcessGroupGloo.cpp:137] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 100 ms
2022-12-01T10:31:02.0992030Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:370: UserWarning: Running monitored_barrier on global rank 2 which does not belong to the given group.
2022-12-01T10:31:02.0992429Z   f"Running {op_name} on global rank {global_rank} which does not "
2022-12-01T10:31:02.0992654Z ok (2.155s)
2022-12-01T10:31:02.0992761Z 
2022-12-01T10:31:02.0992962Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0993195Z Ran 1 test in 2.155s
2022-12-01T10:31:02.0993310Z 
2022-12-01T10:31:02.0993374Z OK
2022-12-01T10:31:02.0993468Z 
2022-12-01T10:31:02.0993555Z Generating XML reports...
2022-12-01T10:31:02.0994006Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102803.xml
2022-12-01T10:31:02.0994507Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.0994977Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0995310Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0995731Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0996081Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0996248Z 
2022-12-01T10:31:02.0996324Z Running tests...
2022-12-01T10:31:02.0996612Z ----------------------------------------------------------------------
2022-12-01T10:31:02.0997032Z   test_monitored_barrier_wait_all_ranks (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26021
2022-12-01T10:31:02.0997449Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26022
2022-12-01T10:31:02.0997788Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 26023
2022-12-01T10:31:02.0998271Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.0998632Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.0999064Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.0999414Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.0999740Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:31:02.1000216Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.1000551Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.1000972Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.1001323Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.1001667Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:31:02.1002168Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.1002657Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.1002989Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.1003419Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.1003764Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.1004089Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:31:02.1004593Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.1005117Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.1005494Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:31:02.1005850Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:31:02.1006197Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:31:02.1006552Z [E ProcessGroupGloo.cpp:2803] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 100 ms
2022-12-01T10:31:02.1006885Z [E ProcessGroupGloo.cpp:2803] [Rank 0]: Rank 2 failed to pass monitoredBarrier in 100 ms
2022-12-01T10:31:02.1007236Z [E ProcessGroupGloo.cpp:137] [Rank 0]: Ranks 1, 2 failed to pass monitoredBarrier in 100 ms
2022-12-01T10:31:02.1007499Z ok (1.954s)
2022-12-01T10:31:02.1007604Z 
2022-12-01T10:31:02.1007808Z ----------------------------------------------------------------------
2022-12-01T10:31:02.1008043Z Ran 1 test in 1.954s
2022-12-01T10:31:02.1008159Z 
2022-12-01T10:31:02.1008222Z OK
2022-12-01T10:31:02.1008316Z 
2022-12-01T10:31:02.1008404Z Generating XML reports...
2022-12-01T10:31:02.1008856Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102807.xml
2022-12-01T10:31:02.1009359Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.1009828Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.1010162Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.1010584Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.1010936Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.1011136Z 
2022-12-01T10:31:02.1011216Z Running tests...
2022-12-01T10:31:02.1011549Z ----------------------------------------------------------------------
2022-12-01T10:31:02.1012023Z   test_nccl_backend_bool_allgather (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl'} (0.002s)
2022-12-01T10:31:02.1012254Z 
2022-12-01T10:31:02.1012453Z ----------------------------------------------------------------------
2022-12-01T10:31:02.1012694Z Ran 1 test in 0.002s
2022-12-01T10:31:02.1012795Z 
2022-12-01T10:31:02.1012869Z OK (skipped=1)
2022-12-01T10:31:02.1012979Z 
2022-12-01T10:31:02.1013064Z Generating XML reports...
2022-12-01T10:31:02.1013526Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102810.xml
2022-12-01T10:31:02.1014008Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.1014477Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.1014817Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.1015340Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.1015678Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.1015845Z 
2022-12-01T10:31:02.1015918Z Running tests...
2022-12-01T10:31:02.1016232Z ----------------------------------------------------------------------
2022-12-01T10:31:02.1016695Z   test_nccl_backend_bool_allreduce (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl'} (0.002s)
2022-12-01T10:31:02.1016926Z 
2022-12-01T10:31:02.1017121Z ----------------------------------------------------------------------
2022-12-01T10:31:02.1017366Z Ran 1 test in 0.002s
2022-12-01T10:31:02.1017482Z 
2022-12-01T10:31:02.1017556Z OK (skipped=1)
2022-12-01T10:31:02.1017667Z 
2022-12-01T10:31:02.1017740Z Generating XML reports...
2022-12-01T10:31:02.1018204Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102811.xml
2022-12-01T10:31:02.1018705Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.1019173Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.1019493Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.1019924Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.1020275Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.1020599Z 
2022-12-01T10:31:02.1020661Z Running tests...
2022-12-01T10:31:02.1020970Z ----------------------------------------------------------------------
2022-12-01T10:31:02.1021451Z   test_nccl_backend_bool_broadcast (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl'} (0.002s)
2022-12-01T10:31:02.1021683Z 
2022-12-01T10:31:02.1021875Z ----------------------------------------------------------------------
2022-12-01T10:31:02.1022102Z Ran 1 test in 0.002s
2022-12-01T10:31:02.1022216Z 
2022-12-01T10:31:02.1022291Z OK (skipped=1)
2022-12-01T10:31:02.1022403Z 
2022-12-01T10:31:02.1022491Z Generating XML reports...
2022-12-01T10:31:02.1022937Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102813.xml
2022-12-01T10:31:02.1023438Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.1023905Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.1024238Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.1024720Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.1025108Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.1025276Z 
2022-12-01T10:31:02.1025353Z Running tests...
2022-12-01T10:31:02.1025644Z ----------------------------------------------------------------------
2022-12-01T10:31:02.1026117Z   test_nccl_backend_bool_reduce (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl'} (0.002s)
2022-12-01T10:31:02.1026344Z 
2022-12-01T10:31:02.1026540Z ----------------------------------------------------------------------
2022-12-01T10:31:02.1026784Z Ran 1 test in 0.002s
2022-12-01T10:31:02.1026898Z 
2022-12-01T10:31:02.1026957Z OK (skipped=1)
2022-12-01T10:31:02.1027066Z 
2022-12-01T10:31:02.1027151Z Generating XML reports...
2022-12-01T10:31:02.1027616Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102814.xml
2022-12-01T10:31:02.1028112Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.1028569Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.1028903Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.1029332Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.1029670Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.1029834Z 
2022-12-01T10:31:02.1029909Z Running tests...
2022-12-01T10:31:02.1030212Z ----------------------------------------------------------------------
2022-12-01T10:31:02.1030613Z   test_nccl_high_priority_stream (__main__.TestDistBackendWithSpawn) ... skip: Only NCCL backend supports high priority stream (0.002s)
2022-12-01T10:31:02.1030851Z 
2022-12-01T10:31:02.1031036Z ----------------------------------------------------------------------
2022-12-01T10:31:02.1031282Z Ran 1 test in 0.002s
2022-12-01T10:31:02.1031396Z 
2022-12-01T10:31:02.1031467Z OK (skipped=1)
2022-12-01T10:31:02.1031582Z 
2022-12-01T10:31:02.1031669Z Generating XML reports...
2022-12-01T10:31:02.1032116Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102815.xml
2022-12-01T10:31:02.1032614Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.1033079Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.1033400Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.1033831Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.1034180Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.1034342Z 
2022-12-01T10:31:02.1034421Z Running tests...
2022-12-01T10:31:02.1034714Z ----------------------------------------------------------------------
2022-12-01T10:31:02.1035062Z   test_new_subgroups (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2022-12-01T10:31:02.1035267Z 
2022-12-01T10:31:02.1035462Z ----------------------------------------------------------------------
2022-12-01T10:31:02.1035691Z Ran 1 test in 0.002s
2022-12-01T10:31:02.1035806Z 
2022-12-01T10:31:02.1035880Z OK (skipped=1)
2022-12-01T10:31:02.1035991Z 
2022-12-01T10:31:02.1036076Z Generating XML reports...
2022-12-01T10:31:02.1036538Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102817.xml
2022-12-01T10:31:02.1037016Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.1037520Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.1037899Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.1038324Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.1038677Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.1038844Z 
2022-12-01T10:31:02.1038918Z Running tests...
2022-12-01T10:31:02.1039223Z ----------------------------------------------------------------------
2022-12-01T10:31:02.1039572Z   test_new_subgroups_by_enumeration (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2022-12-01T10:31:02.1039789Z 
2022-12-01T10:31:02.1039985Z ----------------------------------------------------------------------
2022-12-01T10:31:02.1040230Z Ran 1 test in 0.002s
2022-12-01T10:31:02.1040346Z 
2022-12-01T10:31:02.1040420Z OK (skipped=1)
2022-12-01T10:31:02.1040513Z 
2022-12-01T10:31:02.1040601Z Generating XML reports...
2022-12-01T10:31:02.1041064Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102818.xml
2022-12-01T10:31:02.1041561Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.1042013Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.1042347Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.1042778Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.1043126Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.1043278Z 
2022-12-01T10:31:02.1043354Z Running tests...
2022-12-01T10:31:02.1043660Z ----------------------------------------------------------------------
2022-12-01T10:31:02.1044060Z   test_new_subgroups_by_enumeration_input_rank_exceeds_world_size (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2022-12-01T10:31:02.1044312Z 
2022-12-01T10:31:02.1044508Z ----------------------------------------------------------------------
2022-12-01T10:31:02.1044736Z Ran 1 test in 0.002s
2022-12-01T10:31:02.1044853Z 
2022-12-01T10:31:02.1044928Z OK (skipped=1)
2022-12-01T10:31:02.1045037Z 
2022-12-01T10:31:02.1045124Z Generating XML reports...
2022-12-01T10:31:02.1045569Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102819.xml
2022-12-01T10:31:02.1046060Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.1046524Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.1046857Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.1047279Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.1047630Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.1047797Z 
2022-12-01T10:31:02.1047871Z Running tests...
2022-12-01T10:31:02.1048162Z ----------------------------------------------------------------------
2022-12-01T10:31:02.1048652Z   test_new_subgroups_by_enumeration_negative_input_rank (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26154
2022-12-01T10:31:02.1049089Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26155
2022-12-01T10:31:02.1049430Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 26156
2022-12-01T10:31:02.1049872Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.1050243Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.1050704Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.1051038Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.1051381Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:31:02.1051858Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.1052189Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.1052611Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.1052966Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.1053311Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:31:02.1053787Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.1054108Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.1054540Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.1054886Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.1055212Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:31:02.1055813Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.1056342Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.1056865Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.1057241Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:31:02.1057600Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:31:02.1057949Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:31:02.1058233Z skip: CUDA is not available. (1.751s)
2022-12-01T10:31:02.1058355Z 
2022-12-01T10:31:02.1058555Z ----------------------------------------------------------------------
2022-12-01T10:31:02.1058804Z Ran 1 test in 1.752s
2022-12-01T10:31:02.1058921Z 
2022-12-01T10:31:02.1058994Z OK (skipped=1)
2022-12-01T10:31:02.1059103Z 
2022-12-01T10:31:02.1059175Z Generating XML reports...
2022-12-01T10:31:02.1059633Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102821.xml
2022-12-01T10:31:02.1060129Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.1060743Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.1061066Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.1061497Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.1061850Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.1062018Z 
2022-12-01T10:31:02.1062079Z Running tests...
2022-12-01T10:31:02.1062385Z ----------------------------------------------------------------------
2022-12-01T10:31:02.1062815Z   test_new_subgroups_group_size_exceeds_world_size (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26207
2022-12-01T10:31:02.1063350Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26208
2022-12-01T10:31:02.1063674Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 26209
2022-12-01T10:31:02.1064166Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.1064511Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.1064930Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.1065283Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.1065625Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:31:02.1066101Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.1066419Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.1066851Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.1067199Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.1067536Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:31:02.1067991Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.1068323Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.1068750Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.1069083Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.1069420Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:31:02.1069921Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.1070447Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.1070957Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.1071349Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:31:02.1071699Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:31:02.1072047Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:31:02.1072316Z skip: CUDA is not available. (1.853s)
2022-12-01T10:31:02.1072451Z 
2022-12-01T10:31:02.1072650Z ----------------------------------------------------------------------
2022-12-01T10:31:02.1072895Z Ran 1 test in 1.853s
2022-12-01T10:31:02.1073015Z 
2022-12-01T10:31:02.1073074Z OK (skipped=1)
2022-12-01T10:31:02.1073186Z 
2022-12-01T10:31:02.1073271Z Generating XML reports...
2022-12-01T10:31:02.1073735Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102824.xml
2022-12-01T10:31:02.1074239Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.1074694Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.1075029Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.1075462Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.1075796Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.1075966Z 
2022-12-01T10:31:02.1076040Z Running tests...
2022-12-01T10:31:02.1076383Z ----------------------------------------------------------------------
2022-12-01T10:31:02.1076787Z   test_new_subgroups_overlap_not_allowed (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2022-12-01T10:31:02.1077011Z 
2022-12-01T10:31:02.1077194Z ----------------------------------------------------------------------
2022-12-01T10:31:02.1077440Z Ran 1 test in 0.002s
2022-12-01T10:31:02.1077558Z 
2022-12-01T10:31:02.1077631Z OK (skipped=1)
2022-12-01T10:31:02.1077741Z 
2022-12-01T10:31:02.1077827Z Generating XML reports...
2022-12-01T10:31:02.1078276Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102827.xml
2022-12-01T10:31:02.1078778Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.1079244Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.1079567Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.1080001Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.1080354Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.1080523Z 
2022-12-01T10:31:02.1080597Z Running tests...
2022-12-01T10:31:02.1080889Z ----------------------------------------------------------------------
2022-12-01T10:31:02.1081275Z   test_new_subgroups_world_size_not_divisible_by_group_size (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2022-12-01T10:31:02.1081515Z 
2022-12-01T10:31:02.1081708Z ----------------------------------------------------------------------
2022-12-01T10:31:02.1081937Z Ran 1 test in 0.002s
2022-12-01T10:31:02.1082052Z 
2022-12-01T10:31:02.1082124Z OK (skipped=1)
2022-12-01T10:31:02.1082236Z 
2022-12-01T10:31:02.1082322Z Generating XML reports...
2022-12-01T10:31:02.1082788Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102829.xml
2022-12-01T10:31:02.1083274Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.1083741Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.1084076Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.1084494Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.1084845Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.1084850Z 
2022-12-01T10:31:02.1084926Z Running tests...
2022-12-01T10:31:02.1085123Z ----------------------------------------------------------------------
2022-12-01T10:31:02.1085840Z   test_output_unused_in_loss_dict_module (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78112 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.235s)
2022-12-01T10:31:02.1085848Z 
2022-12-01T10:31:02.1086043Z ----------------------------------------------------------------------
2022-12-01T10:31:02.1086121Z Ran 1 test in 0.235s
2022-12-01T10:31:02.1086126Z 
2022-12-01T10:31:02.1086200Z OK (skipped=1)
2022-12-01T10:31:02.1086205Z 
2022-12-01T10:31:02.1086288Z Generating XML reports...
2022-12-01T10:31:02.1086612Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102830.xml
2022-12-01T10:31:02.1086847Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.1087152Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.1087279Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.1087587Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.1087724Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.1087729Z 
2022-12-01T10:31:02.1087802Z Running tests...
2022-12-01T10:31:02.1087997Z ----------------------------------------------------------------------
2022-12-01T10:31:02.1088248Z   test_output_unused_in_loss_tuple_module (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26290
2022-12-01T10:31:02.1088392Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26291
2022-12-01T10:31:02.1088574Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 26292
2022-12-01T10:31:02.1088881Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.1089007Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.1089290Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.1089425Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.1089603Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:31:02.1089876Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.1089987Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.1090268Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.1090405Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.1090585Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:31:02.1090860Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.1090985Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.1091264Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.1091399Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.1091576Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:31:02.1091865Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.1092168Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.1092463Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.1092634Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:31:02.1092799Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:31:02.1092961Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:31:02.1093068Z skip: Need at least 2 CUDA devices (1.853s)
2022-12-01T10:31:02.1093074Z 
2022-12-01T10:31:02.1093272Z ----------------------------------------------------------------------
2022-12-01T10:31:02.1093348Z Ran 1 test in 1.853s
2022-12-01T10:31:02.1093353Z 
2022-12-01T10:31:02.1093413Z OK (skipped=1)
2022-12-01T10:31:02.1093417Z 
2022-12-01T10:31:02.1093504Z Generating XML reports...
2022-12-01T10:31:02.1093883Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102832.xml
2022-12-01T10:31:02.1094156Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.1094431Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.1094558Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.1094844Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.1094983Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.1094988Z 
2022-12-01T10:31:02.1095062Z Running tests...
2022-12-01T10:31:02.1095320Z ----------------------------------------------------------------------
2022-12-01T10:31:02.1095571Z   test_periodic_model_averager (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26343
2022-12-01T10:31:02.1095732Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26344
2022-12-01T10:31:02.1095895Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 26345
2022-12-01T10:31:02.1096171Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.1096297Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.1096580Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.1096718Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.1096882Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:31:02.1097153Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.1097279Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.1097565Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.1097702Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.1097882Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:31:02.1098182Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.1098451Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.1098575Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.1098841Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.1098979Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.1099159Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:31:02.1099462Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.1099760Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.1099928Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:31:02.1100092Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:31:02.1100255Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:31:02.1100449Z skip: Need at least 2 CUDA devices (1.757s)
2022-12-01T10:31:02.1100456Z 
2022-12-01T10:31:02.1100704Z ----------------------------------------------------------------------
2022-12-01T10:31:02.1100814Z Ran 1 test in 1.757s
2022-12-01T10:31:02.1100819Z 
2022-12-01T10:31:02.1100894Z OK (skipped=1)
2022-12-01T10:31:02.1100898Z 
2022-12-01T10:31:02.1100987Z Generating XML reports...
2022-12-01T10:31:02.1101332Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102835.xml
2022-12-01T10:31:02.1101568Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.1101846Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.1101971Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.1102240Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.1102377Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.1102386Z 
2022-12-01T10:31:02.1102461Z Running tests...
2022-12-01T10:31:02.1102665Z ----------------------------------------------------------------------
2022-12-01T10:31:02.1102917Z   test_periodic_model_averager_param_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26396
2022-12-01T10:31:02.1103075Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26397
2022-12-01T10:31:02.1103234Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 26398
2022-12-01T10:31:02.1103509Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.1103633Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.1103901Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.1104040Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.1104220Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:31:02.1104493Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.1104618Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.1104904Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.1105040Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.1105218Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:31:02.1105491Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.1105599Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.1105886Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.1106024Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.1106200Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:31:02.1106508Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.1106807Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.1107099Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.1107265Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:31:02.1107466Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:31:02.1107643Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:31:02.1107749Z skip: Need at least 2 CUDA devices (1.751s)
2022-12-01T10:31:02.1107754Z 
2022-12-01T10:31:02.1107953Z ----------------------------------------------------------------------
2022-12-01T10:31:02.1108031Z Ran 1 test in 1.751s
2022-12-01T10:31:02.1108036Z 
2022-12-01T10:31:02.1108108Z OK (skipped=1)
2022-12-01T10:31:02.1108113Z 
2022-12-01T10:31:02.1108198Z Generating XML reports...
2022-12-01T10:31:02.1108542Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102838.xml
2022-12-01T10:31:02.1108774Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.1109036Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.1109162Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.1109446Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.1109584Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.1109589Z 
2022-12-01T10:31:02.1109663Z Running tests...
2022-12-01T10:31:02.1109858Z ----------------------------------------------------------------------
2022-12-01T10:31:02.1110561Z   test_post_localSGD_optimizer_parity (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77123 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.232s)
2022-12-01T10:31:02.1110568Z 
2022-12-01T10:31:02.1110766Z ----------------------------------------------------------------------
2022-12-01T10:31:02.1110842Z Ran 1 test in 0.232s
2022-12-01T10:31:02.1110849Z 
2022-12-01T10:31:02.1110921Z OK (skipped=1)
2022-12-01T10:31:02.1110926Z 
2022-12-01T10:31:02.1110996Z Generating XML reports...
2022-12-01T10:31:02.1111339Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102841.xml
2022-12-01T10:31:02.1111573Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.1111851Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.1111975Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.1112259Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.1112395Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.1112400Z 
2022-12-01T10:31:02.1112476Z Running tests...
2022-12-01T10:31:02.1112673Z ----------------------------------------------------------------------
2022-12-01T10:31:02.1113375Z   test_post_localSGD_optimizer_parity_grad_is_view (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77292 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.234s)
2022-12-01T10:31:02.1113397Z 
2022-12-01T10:31:02.1113576Z ----------------------------------------------------------------------
2022-12-01T10:31:02.1113652Z Ran 1 test in 0.234s
2022-12-01T10:31:02.1113657Z 
2022-12-01T10:31:02.1113734Z OK (skipped=1)
2022-12-01T10:31:02.1113738Z 
2022-12-01T10:31:02.1113825Z Generating XML reports...
2022-12-01T10:31:02.1114195Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102843.xml
2022-12-01T10:31:02.1114454Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.1114732Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.1114857Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.1115139Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.1115263Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.1115268Z 
2022-12-01T10:31:02.1115341Z Running tests...
2022-12-01T10:31:02.1115531Z ----------------------------------------------------------------------
2022-12-01T10:31:02.1115806Z   test_post_localSGD_optimizer_parity_with_hierarchical_sgd (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26469
2022-12-01T10:31:02.1115964Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26470
2022-12-01T10:31:02.1116124Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 26471
2022-12-01T10:31:02.1116401Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.1116525Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.1116810Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.1116932Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.1117111Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:31:02.1117382Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.1117510Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.1117795Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.1117928Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.1118105Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:31:02.1118377Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.1118501Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.1118768Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.1118903Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.1119086Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:31:02.1119388Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.1119689Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.1119982Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.1120149Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:31:02.1120313Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:31:02.1120461Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:31:02.1120567Z skip: Need at least 4 CUDA devices (1.853s)
2022-12-01T10:31:02.1120573Z 
2022-12-01T10:31:02.1120802Z ----------------------------------------------------------------------
2022-12-01T10:31:02.1120880Z Ran 1 test in 1.853s
2022-12-01T10:31:02.1120925Z 
2022-12-01T10:31:02.1120998Z OK (skipped=1)
2022-12-01T10:31:02.1121002Z 
2022-12-01T10:31:02.1121087Z Generating XML reports...
2022-12-01T10:31:02.1121430Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102845.xml
2022-12-01T10:31:02.1121660Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.1121936Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.1122048Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.1122330Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.1122467Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.1122475Z 
2022-12-01T10:31:02.1122550Z Running tests...
2022-12-01T10:31:02.1122750Z ----------------------------------------------------------------------
2022-12-01T10:31:02.1123041Z   test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26522
2022-12-01T10:31:02.1123198Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26523
2022-12-01T10:31:02.1123355Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 26524
2022-12-01T10:31:02.1123626Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.1123738Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.1124023Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.1124161Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.1124341Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:31:02.1124611Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.1124735Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.1125018Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.1125152Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.1125330Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:31:02.1125588Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.1125713Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.1125993Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.1126128Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.1126304Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:31:02.1126603Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.1126902Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.1127200Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.1127364Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:31:02.1127543Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:31:02.1127733Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:31:02.1127838Z skip: Need at least 4 CUDA devices (1.753s)
2022-12-01T10:31:02.1127844Z 
2022-12-01T10:31:02.1128043Z ----------------------------------------------------------------------
2022-12-01T10:31:02.1128117Z Ran 1 test in 1.754s
2022-12-01T10:31:02.1128122Z 
2022-12-01T10:31:02.1128191Z OK (skipped=1)
2022-12-01T10:31:02.1128196Z 
2022-12-01T10:31:02.1128280Z Generating XML reports...
2022-12-01T10:31:02.1128621Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102848.xml
2022-12-01T10:31:02.1128840Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.1129114Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.1129240Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.1129524Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.1129658Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.1129663Z 
2022-12-01T10:31:02.1129735Z Running tests...
2022-12-01T10:31:02.1129929Z ----------------------------------------------------------------------
2022-12-01T10:31:02.1130638Z   test_post_localSGD_optimizer_step_reload (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/84886 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.234s)
2022-12-01T10:31:02.1130644Z 
2022-12-01T10:31:02.1130839Z ----------------------------------------------------------------------
2022-12-01T10:31:02.1130915Z Ran 1 test in 0.234s
2022-12-01T10:31:02.1130920Z 
2022-12-01T10:31:02.1130978Z OK (skipped=1)
2022-12-01T10:31:02.1130998Z 
2022-12-01T10:31:02.1131069Z Generating XML reports...
2022-12-01T10:31:02.1131408Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102851.xml
2022-12-01T10:31:02.1131638Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.1131910Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.1132033Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.1132317Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.1132451Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.1132458Z 
2022-12-01T10:31:02.1132529Z Running tests...
2022-12-01T10:31:02.1132711Z ----------------------------------------------------------------------
2022-12-01T10:31:02.1132948Z   test_reduce_full_group_max (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26585
2022-12-01T10:31:02.1133106Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26586
2022-12-01T10:31:02.1133259Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 26587
2022-12-01T10:31:02.1133533Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.1133657Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.1133937Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.1134101Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.1134281Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:31:02.1134566Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.1134692Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.1134970Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.1135104Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.1135361Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:31:02.1135647Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.1135769Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.1136056Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.1136181Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.1136355Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:31:02.1136657Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.1136956Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.1137252Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.1137419Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:31:02.1137586Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:31:02.1137748Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:31:02.1137929Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T10:31:02.1138090Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-12-01T10:31:02.1138263Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T10:31:02.1138563Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:31:02.1138864Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:31:02.1139159Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:31:02.1139410Z STAGE:2022-12-01 10:28:54 26585:26585 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.1139652Z STAGE:2022-12-01 10:28:54 26587:26587 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.1139887Z STAGE:2022-12-01 10:28:54 26586:26586 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.1140133Z STAGE:2022-12-01 10:28:54 26587:26587 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.1140502Z STAGE:2022-12-01 10:28:54 26586:26586 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.1140737Z STAGE:2022-12-01 10:28:54 26585:26585 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.1141141Z STAGE:2022-12-01 10:28:54 26587:26587 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-12-01 10:28:54 26586:26586 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.1141147Z 
2022-12-01T10:31:02.1141449Z STAGE:2022-12-01 10:28:54 26585:26585 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.1141752Z STAGE:2022-12-01 10:28:54 26587:26587 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.1141994Z STAGE:2022-12-01 10:28:54 26586:26586 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.1142234Z STAGE:2022-12-01 10:28:54 26585:26585 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.1142470Z STAGE:2022-12-01 10:28:54 26587:26587 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.1142709Z STAGE:2022-12-01 10:28:54 26585:26585 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.1142944Z STAGE:2022-12-01 10:28:54 26586:26586 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.1143183Z STAGE:2022-12-01 10:28:54 26586:26586 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.1143583Z STAGE:2022-12-01 10:28:54 26585:26585 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 10:28:54 26587:26587 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.1143607Z 
2022-12-01T10:31:02.1143661Z ok (1.955s)
2022-12-01T10:31:02.1143666Z 
2022-12-01T10:31:02.1143862Z ----------------------------------------------------------------------
2022-12-01T10:31:02.1143939Z Ran 1 test in 1.955s
2022-12-01T10:31:02.1143943Z 
2022-12-01T10:31:02.1144003Z OK
2022-12-01T10:31:02.1144008Z 
2022-12-01T10:31:02.1144092Z Generating XML reports...
2022-12-01T10:31:02.1144439Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102853.xml
2022-12-01T10:31:02.1144672Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.1144951Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.1145062Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.1145344Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.1145483Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.1145488Z 
2022-12-01T10:31:02.1145561Z Running tests...
2022-12-01T10:31:02.1145758Z ----------------------------------------------------------------------
2022-12-01T10:31:02.1145998Z   test_reduce_full_group_min (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26650
2022-12-01T10:31:02.1146155Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26651
2022-12-01T10:31:02.1146314Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 26652
2022-12-01T10:31:02.1146591Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.1146704Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.1146985Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.1147121Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.1147298Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:31:02.1147576Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.1147699Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.1147984Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.1148121Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.1148321Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:31:02.1148620Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.1148746Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.1149027Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.1149162Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.1149339Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:31:02.1149641Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.1149943Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.1150237Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.1150393Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:31:02.1150557Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:31:02.1150718Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:31:02.1150891Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T10:31:02.1151065Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T10:31:02.1151240Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-12-01T10:31:02.1151538Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:31:02.1151836Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:31:02.1152130Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:31:02.1152541Z STAGE:2022-12-01 10:28:58 26650:26650 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-12-01 10:28:58 26652:26652 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.1152548Z 
2022-12-01T10:31:02.1152778Z STAGE:2022-12-01 10:28:58 26651:26651 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.1153026Z STAGE:2022-12-01 10:28:58 26651:26651 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.1153270Z STAGE:2022-12-01 10:28:58 26652:26652 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.1153515Z STAGE:2022-12-01 10:28:58 26650:26650 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.1153760Z STAGE:2022-12-01 10:28:58 26651:26651 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.1154002Z STAGE:2022-12-01 10:28:58 26652:26652 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.1154239Z STAGE:2022-12-01 10:28:58 26650:26650 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.1154480Z STAGE:2022-12-01 10:28:58 26652:26652 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.1154719Z STAGE:2022-12-01 10:28:58 26651:26651 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.1154943Z STAGE:2022-12-01 10:28:58 26650:26650 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.1155177Z STAGE:2022-12-01 10:28:58 26652:26652 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.1155441Z STAGE:2022-12-01 10:28:58 26650:26650 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.1155704Z STAGE:2022-12-01 10:28:58 26651:26651 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.1155941Z STAGE:2022-12-01 10:28:58 26651:26651 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.1156177Z STAGE:2022-12-01 10:28:58 26652:26652 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.1156417Z STAGE:2022-12-01 10:28:58 26650:26650 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.1156484Z ok (2.055s)
2022-12-01T10:31:02.1156490Z 
2022-12-01T10:31:02.1156688Z ----------------------------------------------------------------------
2022-12-01T10:31:02.1156750Z Ran 1 test in 2.055s
2022-12-01T10:31:02.1156755Z 
2022-12-01T10:31:02.1156818Z OK
2022-12-01T10:31:02.1156823Z 
2022-12-01T10:31:02.1156907Z Generating XML reports...
2022-12-01T10:31:02.1157254Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102856.xml
2022-12-01T10:31:02.1157490Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.1157766Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.1157891Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.1158177Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.1158299Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.1158318Z 
2022-12-01T10:31:02.1158377Z Running tests...
2022-12-01T10:31:02.1158571Z ----------------------------------------------------------------------
2022-12-01T10:31:02.1158815Z   test_reduce_full_group_product (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26715
2022-12-01T10:31:02.1158977Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26716
2022-12-01T10:31:02.1159130Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 26717
2022-12-01T10:31:02.1159406Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.1159529Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.1159809Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.1159931Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.1160109Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:31:02.1160386Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.1160513Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.1160792Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.1160931Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.1161105Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:31:02.1161407Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.1161680Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.1161790Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.1162095Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.1162233Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.1162436Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:31:02.1162734Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.1163033Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.1163200Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:31:02.1163366Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:31:02.1163526Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:31:02.1163692Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T10:31:02.1163864Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T10:31:02.1164166Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:31:02.1164339Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-12-01T10:31:02.1164631Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:31:02.1164930Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:31:02.1165178Z STAGE:2022-12-01 10:29:01 26715:26715 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.1165583Z STAGE:2022-12-01 10:29:01 26717:26717 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-12-01 10:29:01 26716:26716 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.1165591Z 
2022-12-01T10:31:02.1166001Z STAGE:2022-12-01 10:29:01 26715:26715 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 10:29:01 26717:26717 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.1166006Z 
2022-12-01T10:31:02.1166254Z STAGE:2022-12-01 10:29:01 26716:26716 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.1166655Z STAGE:2022-12-01 10:29:01 26716:26716 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-12-01 10:29:01 26717:26717 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.1166660Z 
2022-12-01T10:31:02.1166903Z STAGE:2022-12-01 10:29:01 26715:26715 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.1167134Z STAGE:2022-12-01 10:29:01 26717:26717 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.1167384Z STAGE:2022-12-01 10:29:01 26715:26715 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.1167626Z STAGE:2022-12-01 10:29:01 26716:26716 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.1167867Z STAGE:2022-12-01 10:29:01 26715:26715 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.1168106Z STAGE:2022-12-01 10:29:01 26717:26717 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.1168348Z STAGE:2022-12-01 10:29:01 26716:26716 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.1168595Z STAGE:2022-12-01 10:29:01 26715:26715 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.1168835Z STAGE:2022-12-01 10:29:01 26716:26716 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.1169078Z STAGE:2022-12-01 10:29:01 26717:26717 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.1169164Z ok (1.955s)
2022-12-01T10:31:02.1169195Z 
2022-12-01T10:31:02.1169394Z ----------------------------------------------------------------------
2022-12-01T10:31:02.1169469Z Ran 1 test in 1.956s
2022-12-01T10:31:02.1169474Z 
2022-12-01T10:31:02.1169534Z OK
2022-12-01T10:31:02.1169539Z 
2022-12-01T10:31:02.1169622Z Generating XML reports...
2022-12-01T10:31:02.1169967Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102900.xml
2022-12-01T10:31:02.1170207Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.1170483Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.1170594Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.1170879Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.1171015Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.1171022Z 
2022-12-01T10:31:02.1171099Z Running tests...
2022-12-01T10:31:02.1171297Z ----------------------------------------------------------------------
2022-12-01T10:31:02.1171532Z   test_reduce_full_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26780
2022-12-01T10:31:02.1171689Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26781
2022-12-01T10:31:02.1171842Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 26782
2022-12-01T10:31:02.1172120Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.1172231Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.1172512Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.1172650Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.1172828Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:31:02.1173102Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.1173225Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.1173504Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.1173638Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.1173816Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:31:02.1174077Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.1174201Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.1174484Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.1174619Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.1174800Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:31:02.1175097Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.1175473Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.1175767Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.1175970Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:31:02.1176122Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:31:02.1176342Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:31:02.1176522Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T10:31:02.1176698Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-12-01T10:31:02.1176998Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:31:02.1177171Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T10:31:02.1177468Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:31:02.1177761Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:31:02.1178014Z STAGE:2022-12-01 10:29:05 26780:26780 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.1178244Z STAGE:2022-12-01 10:29:05 26782:26782 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.1178485Z STAGE:2022-12-01 10:29:05 26781:26781 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.1178730Z STAGE:2022-12-01 10:29:05 26782:26782 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.1178975Z STAGE:2022-12-01 10:29:05 26781:26781 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.1179220Z STAGE:2022-12-01 10:29:05 26780:26780 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.1179457Z STAGE:2022-12-01 10:29:05 26781:26781 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.1179695Z STAGE:2022-12-01 10:29:05 26782:26782 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.1179942Z STAGE:2022-12-01 10:29:05 26780:26780 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.1180188Z STAGE:2022-12-01 10:29:05 26782:26782 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.1180619Z STAGE:2022-12-01 10:29:05 26780:26780 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.1180868Z STAGE:2022-12-01 10:29:05 26781:26781 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.1181110Z STAGE:2022-12-01 10:29:05 26782:26782 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.1181345Z STAGE:2022-12-01 10:29:05 26780:26780 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.1181581Z STAGE:2022-12-01 10:29:05 26781:26781 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.1181829Z STAGE:2022-12-01 10:29:05 26780:26780 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.1182072Z STAGE:2022-12-01 10:29:05 26781:26781 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.1182310Z STAGE:2022-12-01 10:29:05 26782:26782 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.1182382Z ok (1.953s)
2022-12-01T10:31:02.1182388Z 
2022-12-01T10:31:02.1182572Z ----------------------------------------------------------------------
2022-12-01T10:31:02.1182647Z Ran 1 test in 1.953s
2022-12-01T10:31:02.1182653Z 
2022-12-01T10:31:02.1182712Z OK
2022-12-01T10:31:02.1182717Z 
2022-12-01T10:31:02.1182801Z Generating XML reports...
2022-12-01T10:31:02.1183145Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102903.xml
2022-12-01T10:31:02.1183377Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.1183711Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.1183872Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.1184143Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.1184278Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.1184283Z 
2022-12-01T10:31:02.1184359Z Running tests...
2022-12-01T10:31:02.1184555Z ----------------------------------------------------------------------
2022-12-01T10:31:02.1184788Z   test_reduce_group_max (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26845
2022-12-01T10:31:02.1184945Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26846
2022-12-01T10:31:02.1185104Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 26847
2022-12-01T10:31:02.1185381Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.1185508Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.1185775Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.1185912Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.1186087Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:31:02.1186363Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.1186486Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.1186771Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.1186906Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.1187083Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:31:02.1187352Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.1187462Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.1187744Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.1187878Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.1188054Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:31:02.1188355Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.1188658Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.1188952Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.1189120Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:31:02.1189283Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:31:02.1189445Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T10:31:02.1189617Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-12-01T10:31:02.1189778Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:31:02.1189947Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T10:31:02.1190275Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:31:02.1190612Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:31:02.1190904Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:31:02.1191156Z STAGE:2022-12-01 10:29:08 26846:26846 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.1191398Z STAGE:2022-12-01 10:29:08 26847:26847 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.1191635Z STAGE:2022-12-01 10:29:08 26847:26847 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.1191879Z STAGE:2022-12-01 10:29:08 26846:26846 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.1192124Z STAGE:2022-12-01 10:29:08 26847:26847 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.1192366Z STAGE:2022-12-01 10:29:08 26846:26846 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.1192613Z STAGE:2022-12-01 10:29:08 26846:26846 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.1192859Z STAGE:2022-12-01 10:29:08 26847:26847 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.1192927Z ok (2.054s)
2022-12-01T10:31:02.1192932Z 
2022-12-01T10:31:02.1193127Z ----------------------------------------------------------------------
2022-12-01T10:31:02.1193203Z Ran 1 test in 2.054s
2022-12-01T10:31:02.1193208Z 
2022-12-01T10:31:02.1193256Z OK
2022-12-01T10:31:02.1193260Z 
2022-12-01T10:31:02.1193350Z Generating XML reports...
2022-12-01T10:31:02.1193691Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102906.xml
2022-12-01T10:31:02.1193935Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.1194212Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.1194337Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.1194619Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.1194754Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.1194760Z 
2022-12-01T10:31:02.1194820Z Running tests...
2022-12-01T10:31:02.1195015Z ----------------------------------------------------------------------
2022-12-01T10:31:02.1195250Z   test_reduce_group_min (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26906
2022-12-01T10:31:02.1195409Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26907
2022-12-01T10:31:02.1195563Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 26908
2022-12-01T10:31:02.1195842Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.1195966Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.1196251Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.1196386Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.1196550Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:31:02.1196825Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.1196948Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.1197259Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.1197421Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.1197598Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:31:02.1197874Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.1197995Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.1198278Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.1198398Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.1198573Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:31:02.1198879Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.1199180Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.1199472Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.1199639Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:31:02.1199803Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:31:02.1199964Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:31:02.1200139Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T10:31:02.1200298Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T10:31:02.1200472Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-12-01T10:31:02.1200770Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:31:02.1201064Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:31:02.1201315Z STAGE:2022-12-01 10:29:11 26907:26907 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.1201557Z STAGE:2022-12-01 10:29:11 26908:26908 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.1201969Z STAGE:2022-12-01 10:29:11 26907:26907 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 10:29:11 26908:26908 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.1201975Z 
2022-12-01T10:31:02.1202222Z STAGE:2022-12-01 10:29:12 26908:26908 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.1202459Z STAGE:2022-12-01 10:29:12 26907:26907 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.1202712Z STAGE:2022-12-01 10:29:12 26907:26907 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.1202941Z STAGE:2022-12-01 10:29:12 26908:26908 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.1203243Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:31:02.1203314Z ok (1.958s)
2022-12-01T10:31:02.1203319Z 
2022-12-01T10:31:02.1203516Z ----------------------------------------------------------------------
2022-12-01T10:31:02.1203591Z Ran 1 test in 1.958s
2022-12-01T10:31:02.1203597Z 
2022-12-01T10:31:02.1203658Z OK
2022-12-01T10:31:02.1203663Z 
2022-12-01T10:31:02.1203745Z Generating XML reports...
2022-12-01T10:31:02.1204125Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102910.xml
2022-12-01T10:31:02.1204371Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.1204649Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.1204774Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.1205058Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.1205194Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.1205198Z 
2022-12-01T10:31:02.1205271Z Running tests...
2022-12-01T10:31:02.1205470Z ----------------------------------------------------------------------
2022-12-01T10:31:02.1205711Z   test_reduce_group_product (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26967
2022-12-01T10:31:02.1205869Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26968
2022-12-01T10:31:02.1206013Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 26969
2022-12-01T10:31:02.1206287Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.1206413Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.1206695Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.1206829Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.1207007Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:31:02.1207280Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.1207406Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.1207688Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.1207810Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.1207987Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:31:02.1208255Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.1208381Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.1208663Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.1208801Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.1208980Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:31:02.1209279Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.1209580Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.1209859Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.1210026Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:31:02.1210191Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:31:02.1210365Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-12-01T10:31:02.1210543Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T10:31:02.1210743Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:31:02.1210942Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T10:31:02.1211241Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:31:02.1211532Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:31:02.1211809Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:31:02.1212060Z STAGE:2022-12-01 10:29:15 26968:26968 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.1212301Z STAGE:2022-12-01 10:29:15 26969:26969 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.1212556Z STAGE:2022-12-01 10:29:15 26969:26969 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.1212803Z STAGE:2022-12-01 10:29:15 26968:26968 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.1213043Z STAGE:2022-12-01 10:29:15 26969:26969 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.1213278Z STAGE:2022-12-01 10:29:15 26968:26968 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.1213519Z STAGE:2022-12-01 10:29:15 26968:26968 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.1213759Z STAGE:2022-12-01 10:29:15 26969:26969 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.1213814Z ok (1.953s)
2022-12-01T10:31:02.1213819Z 
2022-12-01T10:31:02.1214017Z ----------------------------------------------------------------------
2022-12-01T10:31:02.1214092Z Ran 1 test in 1.953s
2022-12-01T10:31:02.1214097Z 
2022-12-01T10:31:02.1214160Z OK
2022-12-01T10:31:02.1214165Z 
2022-12-01T10:31:02.1214250Z Generating XML reports...
2022-12-01T10:31:02.1214592Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102913.xml
2022-12-01T10:31:02.1214829Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.1215106Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.1215297Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.1215575Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.1215716Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.1215721Z 
2022-12-01T10:31:02.1215794Z Running tests...
2022-12-01T10:31:02.1215992Z ----------------------------------------------------------------------
2022-12-01T10:31:02.1216228Z   test_reduce_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27028
2022-12-01T10:31:02.1216387Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27029
2022-12-01T10:31:02.1216546Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 27030
2022-12-01T10:31:02.1216819Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.1216930Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.1217217Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.1217353Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.1217536Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:31:02.1217850Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.1218002Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.1218284Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.1218419Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.1218599Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:31:02.1218858Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.1218985Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.1219264Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.1219400Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.1219577Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:31:02.1219881Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.1220183Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.1220587Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.1220754Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:31:02.1220906Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:31:02.1221073Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:31:02.1221252Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T10:31:02.1221429Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-12-01T10:31:02.1221602Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T10:31:02.1221909Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:31:02.1222205Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:31:02.1222461Z STAGE:2022-12-01 10:29:18 27029:27029 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.1222758Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:31:02.1223009Z STAGE:2022-12-01 10:29:18 27030:27030 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.1223245Z STAGE:2022-12-01 10:29:18 27030:27030 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.1223494Z STAGE:2022-12-01 10:29:18 27029:27029 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.1223735Z STAGE:2022-12-01 10:29:18 27030:27030 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.1223975Z STAGE:2022-12-01 10:29:18 27029:27029 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.1224218Z STAGE:2022-12-01 10:29:18 27030:27030 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.1224461Z STAGE:2022-12-01 10:29:18 27029:27029 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.1224529Z ok (2.053s)
2022-12-01T10:31:02.1224534Z 
2022-12-01T10:31:02.1224730Z ----------------------------------------------------------------------
2022-12-01T10:31:02.1224793Z Ran 1 test in 2.053s
2022-12-01T10:31:02.1224865Z 
2022-12-01T10:31:02.1224915Z OK
2022-12-01T10:31:02.1224952Z 
2022-12-01T10:31:02.1225039Z Generating XML reports...
2022-12-01T10:31:02.1225386Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102917.xml
2022-12-01T10:31:02.1225621Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.1225900Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.1226025Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.1226312Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.1226448Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.1226453Z 
2022-12-01T10:31:02.1226513Z Running tests...
2022-12-01T10:31:02.1226714Z ----------------------------------------------------------------------
2022-12-01T10:31:02.1226942Z   test_reduce_max (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27089
2022-12-01T10:31:02.1227101Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27090
2022-12-01T10:31:02.1227262Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 27091
2022-12-01T10:31:02.1227533Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.1227659Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.1227942Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.1228077Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.1228244Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:31:02.1228520Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.1228645Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.1228927Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.1229063Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.1229241Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:31:02.1229546Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.1229823Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.1229951Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.1230218Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.1230356Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.1230534Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:31:02.1230840Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.1231139Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.1231305Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:31:02.1231468Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:31:02.1231669Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:31:02.1231924Z STAGE:2022-12-01 10:29:22 27089:27089 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.1232180Z STAGE:2022-12-01 10:29:22 27091:27091 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.1232421Z STAGE:2022-12-01 10:29:22 27090:27090 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.1232835Z STAGE:2022-12-01 10:29:22 27091:27091 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 10:29:22 27090:27090 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.1232841Z 
2022-12-01T10:31:02.1233096Z STAGE:2022-12-01 10:29:22 27089:27089 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.1233500Z STAGE:2022-12-01 10:29:22 27090:27090 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-12-01 10:29:22 27091:27091 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.1233506Z 
2022-12-01T10:31:02.1233754Z STAGE:2022-12-01 10:29:22 27089:27089 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.1234002Z STAGE:2022-12-01 10:29:22 27091:27091 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.1234241Z STAGE:2022-12-01 10:29:22 27089:27089 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.1234486Z STAGE:2022-12-01 10:29:22 27090:27090 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.1234728Z STAGE:2022-12-01 10:29:22 27091:27091 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.1234967Z STAGE:2022-12-01 10:29:22 27089:27089 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.1235207Z STAGE:2022-12-01 10:29:22 27090:27090 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.1235453Z STAGE:2022-12-01 10:29:22 27089:27089 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.1235695Z STAGE:2022-12-01 10:29:22 27090:27090 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.1235919Z STAGE:2022-12-01 10:29:22 27091:27091 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.1235988Z ok (1.854s)
2022-12-01T10:31:02.1235993Z 
2022-12-01T10:31:02.1236191Z ----------------------------------------------------------------------
2022-12-01T10:31:02.1236269Z Ran 1 test in 1.854s
2022-12-01T10:31:02.1236274Z 
2022-12-01T10:31:02.1236336Z OK
2022-12-01T10:31:02.1236340Z 
2022-12-01T10:31:02.1236426Z Generating XML reports...
2022-12-01T10:31:02.1236770Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102920.xml
2022-12-01T10:31:02.1237003Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.1237268Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.1237397Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.1237678Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.1237813Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.1237819Z 
2022-12-01T10:31:02.1237893Z Running tests...
2022-12-01T10:31:02.1238090Z ----------------------------------------------------------------------
2022-12-01T10:31:02.1238317Z   test_reduce_min (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27145
2022-12-01T10:31:02.1238472Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27146
2022-12-01T10:31:02.1238629Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 27147
2022-12-01T10:31:02.1238920Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.1239082Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.1239366Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.1239501Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.1239676Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:31:02.1239947Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.1240070Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.1240350Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.1240485Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.1240652Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:31:02.1240952Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.1241226Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.1241349Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.1241628Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.1241764Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.1242015Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:31:02.1242378Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.1242677Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.1242830Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:31:02.1242995Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:31:02.1243161Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:31:02.1243410Z STAGE:2022-12-01 10:29:25 27145:27145 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.1243650Z STAGE:2022-12-01 10:29:25 27147:27147 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.1243886Z STAGE:2022-12-01 10:29:25 27146:27146 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.1244144Z STAGE:2022-12-01 10:29:25 27146:27146 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.1244393Z STAGE:2022-12-01 10:29:25 27147:27147 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.1244620Z STAGE:2022-12-01 10:29:25 27145:27145 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.1244858Z STAGE:2022-12-01 10:29:25 27146:27146 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.1245096Z STAGE:2022-12-01 10:29:25 27147:27147 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.1245329Z STAGE:2022-12-01 10:29:25 27145:27145 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.1245571Z STAGE:2022-12-01 10:29:25 27147:27147 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.1245811Z STAGE:2022-12-01 10:29:25 27146:27146 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.1246117Z STAGE:2022-12-01 10:29:25 27145:27145 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.1246384Z STAGE:2022-12-01 10:29:25 27147:27147 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.1246621Z STAGE:2022-12-01 10:29:25 27145:27145 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.1246840Z STAGE:2022-12-01 10:29:25 27146:27146 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.1247080Z STAGE:2022-12-01 10:29:25 27145:27145 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.1247321Z STAGE:2022-12-01 10:29:25 27146:27146 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.1247559Z STAGE:2022-12-01 10:29:25 27147:27147 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.1247629Z ok (1.855s)
2022-12-01T10:31:02.1247635Z 
2022-12-01T10:31:02.1247838Z ----------------------------------------------------------------------
2022-12-01T10:31:02.1247913Z Ran 1 test in 1.855s
2022-12-01T10:31:02.1247921Z 
2022-12-01T10:31:02.1247981Z OK
2022-12-01T10:31:02.1247986Z 
2022-12-01T10:31:02.1248070Z Generating XML reports...
2022-12-01T10:31:02.1248402Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102924.xml
2022-12-01T10:31:02.1248635Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.1248914Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.1249036Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.1249317Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.1249453Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.1249458Z 
2022-12-01T10:31:02.1249531Z Running tests...
2022-12-01T10:31:02.1249731Z ----------------------------------------------------------------------
2022-12-01T10:31:02.1249934Z   test_reduce_multigpu (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl backend supports reduce multigpu (0.002s)
2022-12-01T10:31:02.1249938Z 
2022-12-01T10:31:02.1250119Z ----------------------------------------------------------------------
2022-12-01T10:31:02.1250191Z Ran 1 test in 0.002s
2022-12-01T10:31:02.1250195Z 
2022-12-01T10:31:02.1250265Z OK (skipped=1)
2022-12-01T10:31:02.1250270Z 
2022-12-01T10:31:02.1250353Z Generating XML reports...
2022-12-01T10:31:02.1250696Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102927.xml
2022-12-01T10:31:02.1250928Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.1251203Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.1251326Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.1251612Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.1251734Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.1251740Z 
2022-12-01T10:31:02.1251811Z Running tests...
2022-12-01T10:31:02.1252006Z ----------------------------------------------------------------------
2022-12-01T10:31:02.1252235Z   test_reduce_product (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27211
2022-12-01T10:31:02.1252393Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27212
2022-12-01T10:31:02.1252549Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 27213
2022-12-01T10:31:02.1253004Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.1253227Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.1253520Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.1253641Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.1253821Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:31:02.1254094Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.1254217Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.1254496Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.1254631Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.1254814Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:31:02.1255088Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.1255309Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.1255596Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.1255734Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.1255911Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:31:02.1256212Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.1256509Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.1256805Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.1256971Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:31:02.1257139Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:31:02.1257289Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:31:02.1257539Z STAGE:2022-12-01 10:29:30 27211:27211 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.1257780Z STAGE:2022-12-01 10:29:30 27212:27212 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.1258018Z STAGE:2022-12-01 10:29:30 27213:27213 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.1258267Z STAGE:2022-12-01 10:29:30 27213:27213 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.1258509Z STAGE:2022-12-01 10:29:30 27212:27212 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.1258749Z STAGE:2022-12-01 10:29:30 27211:27211 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.1258988Z STAGE:2022-12-01 10:29:30 27213:27213 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.1259223Z STAGE:2022-12-01 10:29:30 27212:27212 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.1259443Z STAGE:2022-12-01 10:29:30 27211:27211 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.1259686Z STAGE:2022-12-01 10:29:30 27213:27213 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.1259926Z STAGE:2022-12-01 10:29:30 27211:27211 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.1271803Z STAGE:2022-12-01 10:29:30 27212:27212 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.1272223Z STAGE:2022-12-01 10:29:30 27213:27213 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.1272531Z STAGE:2022-12-01 10:29:30 27211:27211 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.1272775Z STAGE:2022-12-01 10:29:30 27212:27212 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.1273024Z STAGE:2022-12-01 10:29:30 27211:27211 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.1273270Z STAGE:2022-12-01 10:29:30 27212:27212 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.1273510Z STAGE:2022-12-01 10:29:30 27213:27213 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.1273580Z ok (1.854s)
2022-12-01T10:31:02.1273589Z 
2022-12-01T10:31:02.1273788Z ----------------------------------------------------------------------
2022-12-01T10:31:02.1273852Z Ran 1 test in 1.854s
2022-12-01T10:31:02.1273876Z 
2022-12-01T10:31:02.1273924Z OK
2022-12-01T10:31:02.1273931Z 
2022-12-01T10:31:02.1274016Z Generating XML reports...
2022-12-01T10:31:02.1274363Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102928.xml
2022-12-01T10:31:02.1274598Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.1274878Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.1275005Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.1275290Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.1275427Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.1275433Z 
2022-12-01T10:31:02.1275493Z Running tests...
2022-12-01T10:31:02.1275694Z ----------------------------------------------------------------------
2022-12-01T10:31:02.1275918Z   test_reduce_scatter_tensor_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA reduce_scatter_tensor (0.002s)
2022-12-01T10:31:02.1275923Z 
2022-12-01T10:31:02.1276117Z ----------------------------------------------------------------------
2022-12-01T10:31:02.1276193Z Ran 1 test in 0.002s
2022-12-01T10:31:02.1276197Z 
2022-12-01T10:31:02.1276269Z OK (skipped=1)
2022-12-01T10:31:02.1276274Z 
2022-12-01T10:31:02.1276359Z Generating XML reports...
2022-12-01T10:31:02.1276701Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102932.xml
2022-12-01T10:31:02.1276934Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.1277201Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.1277328Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.1277610Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.1277747Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.1277752Z 
2022-12-01T10:31:02.1277825Z Running tests...
2022-12-01T10:31:02.1278020Z ----------------------------------------------------------------------
2022-12-01T10:31:02.1278226Z   test_reduce_scatter_v_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports reduce_scatter_v (0.002s)
2022-12-01T10:31:02.1278231Z 
2022-12-01T10:31:02.1278424Z ----------------------------------------------------------------------
2022-12-01T10:31:02.1278521Z Ran 1 test in 0.003s
2022-12-01T10:31:02.1278530Z 
2022-12-01T10:31:02.1278614Z OK (skipped=1)
2022-12-01T10:31:02.1278620Z 
2022-12-01T10:31:02.1278751Z Generating XML reports...
2022-12-01T10:31:02.1279441Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102933.xml
2022-12-01T10:31:02.1279731Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.1280007Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.1280131Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.1280417Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.1280553Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.1280559Z 
2022-12-01T10:31:02.1280631Z Running tests...
2022-12-01T10:31:02.1280813Z ----------------------------------------------------------------------
2022-12-01T10:31:02.1281041Z   test_reduce_sum (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27287
2022-12-01T10:31:02.1281201Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27288
2022-12-01T10:31:02.1281358Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 27289
2022-12-01T10:31:02.1281632Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.1281757Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.1282038Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.1282173Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.1282341Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:31:02.1282617Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.1282741Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.1283024Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.1283158Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.1283337Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:31:02.1283607Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.1283729Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.1284007Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.1284131Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.1284309Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:31:02.1284607Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.1284908Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.1285201Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.1285368Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:31:02.1285533Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:31:02.1285695Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:31:02.1286137Z STAGE:2022-12-01 10:29:36 27287:27287 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-12-01 10:29:36 27289:27289 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.1286171Z 
2022-12-01T10:31:02.1286404Z STAGE:2022-12-01 10:29:36 27288:27288 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.1286652Z STAGE:2022-12-01 10:29:36 27289:27289 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.1286894Z STAGE:2022-12-01 10:29:36 27288:27288 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.1287135Z STAGE:2022-12-01 10:29:36 27287:27287 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.1287373Z STAGE:2022-12-01 10:29:36 27289:27289 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.1287609Z STAGE:2022-12-01 10:29:36 27288:27288 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.1287841Z STAGE:2022-12-01 10:29:36 27287:27287 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.1288082Z STAGE:2022-12-01 10:29:36 27289:27289 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.1288321Z STAGE:2022-12-01 10:29:36 27287:27287 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.1288544Z STAGE:2022-12-01 10:29:36 27288:27288 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.1288777Z STAGE:2022-12-01 10:29:36 27289:27289 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.1289010Z STAGE:2022-12-01 10:29:36 27287:27287 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.1289244Z STAGE:2022-12-01 10:29:36 27288:27288 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.1289485Z STAGE:2022-12-01 10:29:36 27287:27287 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.1289722Z STAGE:2022-12-01 10:29:36 27288:27288 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.1289962Z STAGE:2022-12-01 10:29:36 27289:27289 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.1290033Z ok (1.851s)
2022-12-01T10:31:02.1290039Z 
2022-12-01T10:31:02.1290237Z ----------------------------------------------------------------------
2022-12-01T10:31:02.1290300Z Ran 1 test in 1.851s
2022-12-01T10:31:02.1290305Z 
2022-12-01T10:31:02.1290365Z OK
2022-12-01T10:31:02.1290369Z 
2022-12-01T10:31:02.1290453Z Generating XML reports...
2022-12-01T10:31:02.1290797Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102934.xml
2022-12-01T10:31:02.1291029Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.1291300Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.1291425Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.1291708Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.1291836Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.1291855Z 
2022-12-01T10:31:02.1291914Z Running tests...
2022-12-01T10:31:02.1292108Z ----------------------------------------------------------------------
2022-12-01T10:31:02.1292297Z   test_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA reduce (0.002s)
2022-12-01T10:31:02.1292302Z 
2022-12-01T10:31:02.1292496Z ----------------------------------------------------------------------
2022-12-01T10:31:02.1292571Z Ran 1 test in 0.002s
2022-12-01T10:31:02.1292576Z 
2022-12-01T10:31:02.1292647Z OK (skipped=1)
2022-12-01T10:31:02.1292652Z 
2022-12-01T10:31:02.1292736Z Generating XML reports...
2022-12-01T10:31:02.1293075Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102938.xml
2022-12-01T10:31:02.1293326Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.1293629Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.1293753Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.1294035Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.1294171Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.1294177Z 
2022-12-01T10:31:02.1294249Z Running tests...
2022-12-01T10:31:02.1294444Z ----------------------------------------------------------------------
2022-12-01T10:31:02.1294637Z   test_reduce_sum_cuda_twice (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA reduce (0.002s)
2022-12-01T10:31:02.1294642Z 
2022-12-01T10:31:02.1294836Z ----------------------------------------------------------------------
2022-12-01T10:31:02.1294936Z Ran 1 test in 0.002s
2022-12-01T10:31:02.1294943Z 
2022-12-01T10:31:02.1295013Z OK (skipped=1)
2022-12-01T10:31:02.1295018Z 
2022-12-01T10:31:02.1295102Z Generating XML reports...
2022-12-01T10:31:02.1295521Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102939.xml
2022-12-01T10:31:02.1295759Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.1296036Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.1296161Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.1296442Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.1296580Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.1296589Z 
2022-12-01T10:31:02.1296649Z Running tests...
2022-12-01T10:31:02.1296846Z ----------------------------------------------------------------------
2022-12-01T10:31:02.1297077Z   test_reduce_sum_twice (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27363
2022-12-01T10:31:02.1297235Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27364
2022-12-01T10:31:02.1297390Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 27365
2022-12-01T10:31:02.1297662Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.1297786Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.1298067Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.1298190Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.1298369Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:31:02.1298647Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.1298769Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.1299048Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.1299181Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.1299362Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:31:02.1299634Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.1299760Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.1300089Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.1300266Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.1300549Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:31:02.1300857Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.1301155Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.1301450Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.1301615Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:31:02.1301782Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:31:02.1301945Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:31:02.1302185Z STAGE:2022-12-01 10:29:42 27363:27363 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.1302430Z STAGE:2022-12-01 10:29:42 27364:27364 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.1302665Z STAGE:2022-12-01 10:29:42 27365:27365 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.1302914Z STAGE:2022-12-01 10:29:42 27365:27365 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.1303329Z STAGE:2022-12-01 10:29:42 27363:27363 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 10:29:42 27364:27364 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.1303335Z 
2022-12-01T10:31:02.1303579Z STAGE:2022-12-01 10:29:42 27365:27365 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.1303817Z STAGE:2022-12-01 10:29:42 27364:27364 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.1304055Z STAGE:2022-12-01 10:29:42 27363:27363 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.1304302Z STAGE:2022-12-01 10:29:42 27365:27365 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.1304533Z STAGE:2022-12-01 10:29:42 27363:27363 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.1304773Z STAGE:2022-12-01 10:29:42 27364:27364 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.1305170Z STAGE:2022-12-01 10:29:42 27365:27365 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-12-01 10:29:42 27363:27363 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.1305176Z 
2022-12-01T10:31:02.1305417Z STAGE:2022-12-01 10:29:42 27364:27364 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.1305660Z STAGE:2022-12-01 10:29:42 27363:27363 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.1306070Z STAGE:2022-12-01 10:29:42 27365:27365 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 10:29:42 27364:27364 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.1306075Z 
2022-12-01T10:31:02.1306143Z ok (1.853s)
2022-12-01T10:31:02.1306148Z 
2022-12-01T10:31:02.1306343Z ----------------------------------------------------------------------
2022-12-01T10:31:02.1306419Z Ran 1 test in 1.853s
2022-12-01T10:31:02.1306424Z 
2022-12-01T10:31:02.1306483Z OK
2022-12-01T10:31:02.1306488Z 
2022-12-01T10:31:02.1306570Z Generating XML reports...
2022-12-01T10:31:02.1306896Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102940.xml
2022-12-01T10:31:02.1307183Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.1307493Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.1307616Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.1307900Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.1308038Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.1308043Z 
2022-12-01T10:31:02.1308129Z Running tests...
2022-12-01T10:31:02.1308325Z ----------------------------------------------------------------------
2022-12-01T10:31:02.1308537Z   test_scatter (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27419
2022-12-01T10:31:02.1308694Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27420
2022-12-01T10:31:02.1308853Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 27421
2022-12-01T10:31:02.1309128Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.1309254Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.1309535Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.1309669Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.1309847Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:31:02.1310121Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.1310230Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.1310515Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.1310652Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.1310829Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:31:02.1311102Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.1311224Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.1311504Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.1311640Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.1311803Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:31:02.1312104Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.1312401Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.1312697Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.1312861Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:31:02.1313108Z STAGE:2022-12-01 10:29:45 27420:27420 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.1313270Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:31:02.1313429Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:31:02.1313864Z STAGE:2022-12-01 10:29:45 27419:27419 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-12-01 10:29:45 27421:27421 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.1313872Z 
2022-12-01T10:31:02.1314150Z STAGE:2022-12-01 10:29:45 27421:27421 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.1314381Z STAGE:2022-12-01 10:29:45 27420:27420 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.1314621Z STAGE:2022-12-01 10:29:45 27419:27419 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.1314868Z STAGE:2022-12-01 10:29:45 27421:27421 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.1315102Z STAGE:2022-12-01 10:29:45 27420:27420 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.1315337Z STAGE:2022-12-01 10:29:45 27419:27419 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.1315577Z STAGE:2022-12-01 10:29:45 27421:27421 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.1315818Z STAGE:2022-12-01 10:29:45 27419:27419 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.1316056Z STAGE:2022-12-01 10:29:45 27420:27420 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.1316293Z STAGE:2022-12-01 10:29:45 27421:27421 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.1316513Z STAGE:2022-12-01 10:29:45 27420:27420 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.1316743Z STAGE:2022-12-01 10:29:45 27419:27419 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.1316981Z STAGE:2022-12-01 10:29:45 27420:27420 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.1317217Z STAGE:2022-12-01 10:29:45 27421:27421 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.1317453Z STAGE:2022-12-01 10:29:45 27419:27419 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.1317520Z ok (1.954s)
2022-12-01T10:31:02.1317528Z 
2022-12-01T10:31:02.1317727Z ----------------------------------------------------------------------
2022-12-01T10:31:02.1317804Z Ran 1 test in 1.954s
2022-12-01T10:31:02.1317810Z 
2022-12-01T10:31:02.1317870Z OK
2022-12-01T10:31:02.1317874Z 
2022-12-01T10:31:02.1317946Z Generating XML reports...
2022-12-01T10:31:02.1318286Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102944.xml
2022-12-01T10:31:02.1318517Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.1318793Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.1318917Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.1319198Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.1319337Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.1319344Z 
2022-12-01T10:31:02.1319415Z Running tests...
2022-12-01T10:31:02.1319598Z ----------------------------------------------------------------------
2022-12-01T10:31:02.1319827Z   test_scatter_checks (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27475
2022-12-01T10:31:02.1319984Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27476
2022-12-01T10:31:02.1320138Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 27477
2022-12-01T10:31:02.1320413Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.1320535Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.1320816Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.1320979Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.1321185Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:31:02.1321445Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.1321570Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.1321849Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.1321984Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.1322160Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:31:02.1322427Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.1322554Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.1322833Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.1322954Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.1323130Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:31:02.1323427Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.1323722Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.1324014Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.1324180Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:31:02.1324346Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:31:02.1324509Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:31:02.1324577Z ok (1.755s)
2022-12-01T10:31:02.1324582Z 
2022-12-01T10:31:02.1324766Z ----------------------------------------------------------------------
2022-12-01T10:31:02.1324841Z Ran 1 test in 1.755s
2022-12-01T10:31:02.1324846Z 
2022-12-01T10:31:02.1324908Z OK
2022-12-01T10:31:02.1324913Z 
2022-12-01T10:31:02.1324995Z Generating XML reports...
2022-12-01T10:31:02.1325337Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102947.xml
2022-12-01T10:31:02.1325569Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.1325844Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.1325969Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.1326249Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.1326369Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.1326374Z 
2022-12-01T10:31:02.1326449Z Running tests...
2022-12-01T10:31:02.1326644Z ----------------------------------------------------------------------
2022-12-01T10:31:02.1326873Z   test_scatter_complex (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27528
2022-12-01T10:31:02.1327031Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27529
2022-12-01T10:31:02.1327187Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 27530
2022-12-01T10:31:02.1327486Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.1327612Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.1327905Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.1328039Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.1328216Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:31:02.1328485Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.1328606Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.1328883Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.1329018Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.1329198Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:31:02.1329470Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.1329578Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.1329855Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.1329988Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.1330163Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:31:02.1330462Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.1330762Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.1331057Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.1331225Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:31:02.1331472Z STAGE:2022-12-01 10:29:52 27530:27530 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.1331622Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:31:02.1331787Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:31:02.1332030Z STAGE:2022-12-01 10:29:52 27529:27529 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.1332268Z STAGE:2022-12-01 10:29:52 27528:27528 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.1332515Z STAGE:2022-12-01 10:29:52 27529:27529 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.1332758Z STAGE:2022-12-01 10:29:52 27528:27528 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.1333000Z STAGE:2022-12-01 10:29:52 27530:27530 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.1333238Z STAGE:2022-12-01 10:29:52 27529:27529 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.1333636Z STAGE:2022-12-01 10:29:52 27528:27528 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-12-01 10:29:52 27530:27530 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.1333642Z 
2022-12-01T10:31:02.1333869Z STAGE:2022-12-01 10:29:52 27528:27528 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.1334110Z STAGE:2022-12-01 10:29:52 27529:27529 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.1334346Z STAGE:2022-12-01 10:29:52 27530:27530 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.1334621Z STAGE:2022-12-01 10:29:52 27529:27529 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.1334886Z STAGE:2022-12-01 10:29:52 27528:27528 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.1335118Z STAGE:2022-12-01 10:29:52 27530:27530 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.1335457Z STAGE:2022-12-01 10:29:52 27528:27528 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.1335703Z STAGE:2022-12-01 10:29:52 27530:27530 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.1335941Z STAGE:2022-12-01 10:29:52 27529:27529 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.1335995Z ok (1.855s)
2022-12-01T10:31:02.1336013Z 
2022-12-01T10:31:02.1336197Z ----------------------------------------------------------------------
2022-12-01T10:31:02.1336274Z Ran 1 test in 1.855s
2022-12-01T10:31:02.1336279Z 
2022-12-01T10:31:02.1336346Z OK
2022-12-01T10:31:02.1336351Z 
2022-12-01T10:31:02.1336437Z Generating XML reports...
2022-12-01T10:31:02.1336780Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102950.xml
2022-12-01T10:31:02.1337015Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.1337293Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.1337417Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.1337683Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.1337819Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.1337824Z 
2022-12-01T10:31:02.1337897Z Running tests...
2022-12-01T10:31:02.1338096Z ----------------------------------------------------------------------
2022-12-01T10:31:02.1338280Z   test_scatter_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA gather (0.002s)
2022-12-01T10:31:02.1338287Z 
2022-12-01T10:31:02.1338483Z ----------------------------------------------------------------------
2022-12-01T10:31:02.1338558Z Ran 1 test in 0.002s
2022-12-01T10:31:02.1338563Z 
2022-12-01T10:31:02.1338634Z OK (skipped=1)
2022-12-01T10:31:02.1338639Z 
2022-12-01T10:31:02.1338721Z Generating XML reports...
2022-12-01T10:31:02.1339047Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102953.xml
2022-12-01T10:31:02.1339281Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.1339553Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.1339677Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.1339957Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.1340093Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.1340098Z 
2022-12-01T10:31:02.1340170Z Running tests...
2022-12-01T10:31:02.1340474Z ----------------------------------------------------------------------
2022-12-01T10:31:02.1340657Z   test_scatter_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA gather (0.002s)
2022-12-01T10:31:02.1340675Z 
2022-12-01T10:31:02.1340855Z ----------------------------------------------------------------------
2022-12-01T10:31:02.1340929Z Ran 1 test in 0.002s
2022-12-01T10:31:02.1340934Z 
2022-12-01T10:31:02.1341004Z OK (skipped=1)
2022-12-01T10:31:02.1341009Z 
2022-12-01T10:31:02.1341092Z Generating XML reports...
2022-12-01T10:31:02.1341487Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102955.xml
2022-12-01T10:31:02.1341724Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.1342037Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.1342161Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.1342432Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.1342567Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.1342572Z 
2022-12-01T10:31:02.1342645Z Running tests...
2022-12-01T10:31:02.1342846Z ----------------------------------------------------------------------
2022-12-01T10:31:02.1343081Z   test_scatter_full_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27604
2022-12-01T10:31:02.1343240Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27605
2022-12-01T10:31:02.1343399Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 27606
2022-12-01T10:31:02.1343674Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.1343797Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.1344064Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.1344200Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.1344379Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:31:02.1344652Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.1344776Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.1345054Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.1345190Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.1345367Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:31:02.1345623Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.1345746Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.1346025Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.1346160Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.1346339Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:31:02.1346638Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.1346939Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.1347230Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.1347394Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:31:02.1347538Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:31:02.1347700Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:31:02.1347874Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T10:31:02.1348080Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T10:31:02.1348375Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:31:02.1348587Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-12-01T10:31:02.1348887Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:31:02.1349179Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:31:02.1349428Z STAGE:2022-12-01 10:29:58 27604:27604 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.1349819Z STAGE:2022-12-01 10:29:58 27606:27606 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-12-01 10:29:58 27605:27605 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.1349837Z 
2022-12-01T10:31:02.1350074Z STAGE:2022-12-01 10:29:58 27604:27604 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.1350319Z STAGE:2022-12-01 10:29:58 27605:27605 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.1350557Z STAGE:2022-12-01 10:29:58 27606:27606 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.1350799Z STAGE:2022-12-01 10:29:58 27604:27604 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.1351036Z STAGE:2022-12-01 10:29:58 27606:27606 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.1351271Z STAGE:2022-12-01 10:29:58 27605:27605 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.1351512Z STAGE:2022-12-01 10:29:58 27604:27604 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.1351927Z STAGE:2022-12-01 10:29:58 27606:27606 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 10:29:58 27605:27605 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.1351934Z 
2022-12-01T10:31:02.1352177Z STAGE:2022-12-01 10:29:58 27604:27604 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.1352416Z STAGE:2022-12-01 10:29:58 27605:27605 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.1352639Z STAGE:2022-12-01 10:29:58 27606:27606 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.1352883Z STAGE:2022-12-01 10:29:58 27604:27604 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.1353124Z STAGE:2022-12-01 10:29:58 27605:27605 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.1353361Z STAGE:2022-12-01 10:29:58 27606:27606 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.1353429Z ok (1.955s)
2022-12-01T10:31:02.1353434Z 
2022-12-01T10:31:02.1353630Z ----------------------------------------------------------------------
2022-12-01T10:31:02.1353708Z Ran 1 test in 1.955s
2022-12-01T10:31:02.1353713Z 
2022-12-01T10:31:02.1353773Z OK
2022-12-01T10:31:02.1353777Z 
2022-12-01T10:31:02.1353849Z Generating XML reports...
2022-12-01T10:31:02.1354191Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102956.xml
2022-12-01T10:31:02.1354422Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.1354697Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.1354820Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.1355100Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.1355236Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.1355270Z 
2022-12-01T10:31:02.1355344Z Running tests...
2022-12-01T10:31:02.1355568Z ----------------------------------------------------------------------
2022-12-01T10:31:02.1355782Z   test_scatter_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27669
2022-12-01T10:31:02.1355939Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27670
2022-12-01T10:31:02.1356091Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 27671
2022-12-01T10:31:02.1356366Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.1356488Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.1356767Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.1356903Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.1357079Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:31:02.1357343Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.1357466Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.1357745Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.1357881Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.1358057Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:31:02.1358330Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.1358454Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.1358736Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.1358874Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.1359039Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:31:02.1359334Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.1359628Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.1359917Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.1360080Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:31:02.1360245Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:31:02.1360405Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:31:02.1360581Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T10:31:02.1360756Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-12-01T10:31:02.1360915Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T10:31:02.1361208Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:31:02.1361498Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:31:02.1361746Z STAGE:2022-12-01 10:30:01 27670:27670 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.1362067Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:31:02.1362342Z STAGE:2022-12-01 10:30:01 27671:27671 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.1362590Z STAGE:2022-12-01 10:30:01 27671:27671 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.1362834Z STAGE:2022-12-01 10:30:01 27670:27670 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.1363075Z STAGE:2022-12-01 10:30:01 27671:27671 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.1363298Z STAGE:2022-12-01 10:30:01 27670:27670 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.1363540Z STAGE:2022-12-01 10:30:01 27670:27670 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.1363857Z STAGE:2022-12-01 10:30:01 27671:27671 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.1363961Z ok (1.954s)
2022-12-01T10:31:02.1363969Z 
2022-12-01T10:31:02.1364179Z ----------------------------------------------------------------------
2022-12-01T10:31:02.1364256Z Ran 1 test in 1.954s
2022-12-01T10:31:02.1364261Z 
2022-12-01T10:31:02.1364323Z OK
2022-12-01T10:31:02.1364328Z 
2022-12-01T10:31:02.1364416Z Generating XML reports...
2022-12-01T10:31:02.1364755Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103000.xml
2022-12-01T10:31:02.1364975Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.1365249Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.1365373Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.1365653Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.1365790Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.1365797Z 
2022-12-01T10:31:02.1365870Z Running tests...
2022-12-01T10:31:02.1366062Z ----------------------------------------------------------------------
2022-12-01T10:31:02.1366297Z   test_scatter_object_list (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27730
2022-12-01T10:31:02.1366440Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27731
2022-12-01T10:31:02.1366594Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 27732
2022-12-01T10:31:02.1366868Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.1366992Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.1367273Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.1367409Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.1367588Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:31:02.1367861Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.1367983Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.1368247Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.1368380Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.1368554Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:31:02.1368824Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.1369000Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.1369311Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.1369446Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.1369622Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:31:02.1369922Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.1370204Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.1370495Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.1370665Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:31:02.1370828Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:31:02.1370995Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:31:02.1371064Z ok (1.755s)
2022-12-01T10:31:02.1371070Z 
2022-12-01T10:31:02.1371272Z ----------------------------------------------------------------------
2022-12-01T10:31:02.1371347Z Ran 1 test in 1.755s
2022-12-01T10:31:02.1371352Z 
2022-12-01T10:31:02.1371399Z OK
2022-12-01T10:31:02.1371417Z 
2022-12-01T10:31:02.1371488Z Generating XML reports...
2022-12-01T10:31:02.1371830Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103003.xml
2022-12-01T10:31:02.1372063Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.1372339Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.1372462Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.1372742Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.1372877Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.1372883Z 
2022-12-01T10:31:02.1372954Z Running tests...
2022-12-01T10:31:02.1373138Z ----------------------------------------------------------------------
2022-12-01T10:31:02.1373358Z   test_send_recv (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27783
2022-12-01T10:31:02.1373511Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27784
2022-12-01T10:31:02.1373665Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 27785
2022-12-01T10:31:02.1373943Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.1374069Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.1374349Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.1374545Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.1374774Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:31:02.1375154Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.1375357Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.1375717Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.1375857Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.1376090Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:31:02.1376396Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.1376520Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.1376799Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.1376920Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.1377096Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:31:02.1377397Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.1377695Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.1377991Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.1378158Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:31:02.1378322Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:31:02.1378485Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:31:02.1378553Z ok (1.854s)
2022-12-01T10:31:02.1378559Z 
2022-12-01T10:31:02.1378745Z ----------------------------------------------------------------------
2022-12-01T10:31:02.1378820Z Ran 1 test in 1.854s
2022-12-01T10:31:02.1378826Z 
2022-12-01T10:31:02.1378885Z OK
2022-12-01T10:31:02.1378890Z 
2022-12-01T10:31:02.1378974Z Generating XML reports...
2022-12-01T10:31:02.1379316Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103006.xml
2022-12-01T10:31:02.1379550Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.1379828Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.1379952Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.1380218Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.1380446Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.1380454Z 
2022-12-01T10:31:02.1380531Z Running tests...
2022-12-01T10:31:02.1380729Z ----------------------------------------------------------------------
2022-12-01T10:31:02.1380964Z   test_send_recv_any_source (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27836
2022-12-01T10:31:02.1381122Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27837
2022-12-01T10:31:02.1381281Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 27838
2022-12-01T10:31:02.1381558Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.1381683Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.1381950Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.1382085Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.1382261Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:31:02.1382535Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.1382658Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.1383004Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.1383174Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.1383358Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:31:02.1383633Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.1383742Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.1384023Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.1384158Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.1384336Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:31:02.1384640Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.1384942Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.1385235Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.1385400Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:31:02.1385566Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:31:02.1385714Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:31:02.1385784Z ok (1.853s)
2022-12-01T10:31:02.1385789Z 
2022-12-01T10:31:02.1385986Z ----------------------------------------------------------------------
2022-12-01T10:31:02.1386063Z Ran 1 test in 1.853s
2022-12-01T10:31:02.1386070Z 
2022-12-01T10:31:02.1386131Z OK
2022-12-01T10:31:02.1386139Z 
2022-12-01T10:31:02.1386223Z Generating XML reports...
2022-12-01T10:31:02.1386562Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103009.xml
2022-12-01T10:31:02.1386796Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.1387057Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.1387182Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.1387463Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.1387597Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.1387603Z 
2022-12-01T10:31:02.1387676Z Running tests...
2022-12-01T10:31:02.1387875Z ----------------------------------------------------------------------
2022-12-01T10:31:02.1388131Z   test_send_recv_any_source_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27889
2022-12-01T10:31:02.1388286Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27890
2022-12-01T10:31:02.1388439Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 27891
2022-12-01T10:31:02.1388699Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.1388822Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.1389103Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.1389238Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.1389446Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:31:02.1389747Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.1389870Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.1390151Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.1390273Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.1390453Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:31:02.1390725Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.1390847Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.1391128Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.1391261Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.1391439Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:31:02.1391741Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.1392039Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.1392318Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.1392483Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:31:02.1392731Z STAGE:2022-12-01 10:30:14 27890:27890 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.1392896Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:31:02.1393064Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:31:02.1393311Z STAGE:2022-12-01 10:30:14 27889:27889 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.1393556Z STAGE:2022-12-01 10:30:14 27891:27891 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.1393806Z STAGE:2022-12-01 10:30:14 27889:27889 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.1394049Z STAGE:2022-12-01 10:30:14 27890:27890 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.1394275Z STAGE:2022-12-01 10:30:14 27891:27891 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.1394344Z ok (1.956s)
2022-12-01T10:31:02.1394349Z 
2022-12-01T10:31:02.1394547Z ----------------------------------------------------------------------
2022-12-01T10:31:02.1394625Z Ran 1 test in 1.956s
2022-12-01T10:31:02.1394630Z 
2022-12-01T10:31:02.1394693Z OK
2022-12-01T10:31:02.1394698Z 
2022-12-01T10:31:02.1394782Z Generating XML reports...
2022-12-01T10:31:02.1395125Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103013.xml
2022-12-01T10:31:02.1395362Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.1395623Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.1395747Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.1396029Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.1396164Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.1396169Z 
2022-12-01T10:31:02.1396242Z Running tests...
2022-12-01T10:31:02.1396474Z ----------------------------------------------------------------------
2022-12-01T10:31:02.1396768Z   test_send_recv_any_source_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27945
2022-12-01T10:31:02.1396927Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27946
2022-12-01T10:31:02.1397082Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 27947
2022-12-01T10:31:02.1397342Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.1397463Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.1397742Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.1397874Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.1398055Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:31:02.1398355Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.1398627Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.1398750Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.1399028Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.1399149Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.1399324Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:31:02.1399598Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.1399723Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.1400004Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.1400137Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.1400314Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:31:02.1400609Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.1400904Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.1401055Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:31:02.1401218Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:31:02.1401467Z STAGE:2022-12-01 10:30:18 27946:27946 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.1401711Z STAGE:2022-12-01 10:30:18 27947:27947 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.1401872Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:31:02.1402114Z STAGE:2022-12-01 10:30:18 27945:27945 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.1402364Z STAGE:2022-12-01 10:30:18 27945:27945 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.1402604Z STAGE:2022-12-01 10:30:18 27946:27946 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.1402830Z STAGE:2022-12-01 10:30:18 27947:27947 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.1402896Z ok (1.954s)
2022-12-01T10:31:02.1402901Z 
2022-12-01T10:31:02.1403129Z ----------------------------------------------------------------------
2022-12-01T10:31:02.1403205Z Ran 1 test in 1.954s
2022-12-01T10:31:02.1403276Z 
2022-12-01T10:31:02.1403337Z OK
2022-12-01T10:31:02.1403342Z 
2022-12-01T10:31:02.1403425Z Generating XML reports...
2022-12-01T10:31:02.1403766Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103016.xml
2022-12-01T10:31:02.1403998Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.1404275Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.1404386Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.1404667Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.1404803Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.1404808Z 
2022-12-01T10:31:02.1404882Z Running tests...
2022-12-01T10:31:02.1405079Z ----------------------------------------------------------------------
2022-12-01T10:31:02.1405325Z   test_send_recv_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28001
2022-12-01T10:31:02.1405483Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28002
2022-12-01T10:31:02.1405637Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 28003
2022-12-01T10:31:02.1405912Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.1406022Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.1406303Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.1406437Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.1406615Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:31:02.1406890Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.1407014Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.1407297Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.1407433Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.1407597Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:31:02.1407895Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.1408169Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.1408291Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.1408672Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.1408920Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.1409250Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:31:02.1409651Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.1409952Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.1410477Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:31:02.1410703Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:31:02.1410872Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:31:02.1411154Z STAGE:2022-12-01 10:30:21 28001:28001 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.1411396Z STAGE:2022-12-01 10:30:21 28003:28003 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.1411634Z STAGE:2022-12-01 10:30:21 28002:28002 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.1411882Z STAGE:2022-12-01 10:30:21 28003:28003 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.1412127Z STAGE:2022-12-01 10:30:21 28001:28001 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.1412367Z STAGE:2022-12-01 10:30:21 28002:28002 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.1412421Z ok (1.751s)
2022-12-01T10:31:02.1412428Z 
2022-12-01T10:31:02.1412627Z ----------------------------------------------------------------------
2022-12-01T10:31:02.1412705Z Ran 1 test in 1.751s
2022-12-01T10:31:02.1412711Z 
2022-12-01T10:31:02.1412772Z OK
2022-12-01T10:31:02.1412777Z 
2022-12-01T10:31:02.1412860Z Generating XML reports...
2022-12-01T10:31:02.1413203Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103020.xml
2022-12-01T10:31:02.1413435Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.1413706Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.1413818Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.1414097Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.1414231Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.1414238Z 
2022-12-01T10:31:02.1414310Z Running tests...
2022-12-01T10:31:02.1414508Z ----------------------------------------------------------------------
2022-12-01T10:31:02.1414679Z   test_send_recv_nccl (__main__.TestDistBackendWithSpawn) ... skip: NCCL Send Recv Only (0.001s)
2022-12-01T10:31:02.1414683Z 
2022-12-01T10:31:02.1414874Z ----------------------------------------------------------------------
2022-12-01T10:31:02.1414948Z Ran 1 test in 0.002s
2022-12-01T10:31:02.1414953Z 
2022-12-01T10:31:02.1415023Z OK (skipped=1)
2022-12-01T10:31:02.1415028Z 
2022-12-01T10:31:02.1415099Z Generating XML reports...
2022-12-01T10:31:02.1415507Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103023.xml
2022-12-01T10:31:02.1415742Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.1416017Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.1416140Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.1416421Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.1416557Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.1416562Z 
2022-12-01T10:31:02.1416634Z Running tests...
2022-12-01T10:31:02.1416828Z ----------------------------------------------------------------------
2022-12-01T10:31:02.1417008Z   test_send_recv_nccl_autograd_profiler (__main__.TestDistBackendWithSpawn) ... skip: NCCL Send Recv Only (0.002s)
2022-12-01T10:31:02.1417013Z 
2022-12-01T10:31:02.1417207Z ----------------------------------------------------------------------
2022-12-01T10:31:02.1417283Z Ran 1 test in 0.002s
2022-12-01T10:31:02.1417288Z 
2022-12-01T10:31:02.1417360Z OK (skipped=1)
2022-12-01T10:31:02.1417364Z 
2022-12-01T10:31:02.1417449Z Generating XML reports...
2022-12-01T10:31:02.1417829Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103024.xml
2022-12-01T10:31:02.1418089Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.1418364Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.1418475Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.1418755Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.1418889Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.1418894Z 
2022-12-01T10:31:02.1418965Z Running tests...
2022-12-01T10:31:02.1419159Z ----------------------------------------------------------------------
2022-12-01T10:31:02.1419351Z   test_send_recv_nccl_torch_profiler (__main__.TestDistBackendWithSpawn) ... skip: NCCL Send Recv Only (0.002s)
2022-12-01T10:31:02.1419359Z 
2022-12-01T10:31:02.1419549Z ----------------------------------------------------------------------
2022-12-01T10:31:02.1419625Z Ran 1 test in 0.002s
2022-12-01T10:31:02.1419630Z 
2022-12-01T10:31:02.1419701Z OK (skipped=1)
2022-12-01T10:31:02.1419706Z 
2022-12-01T10:31:02.1419775Z Generating XML reports...
2022-12-01T10:31:02.1420110Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103025.xml
2022-12-01T10:31:02.1420341Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.1420724Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.1420848Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.1421133Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.1421268Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.1421275Z 
2022-12-01T10:31:02.1421347Z Running tests...
2022-12-01T10:31:02.1421542Z ----------------------------------------------------------------------
2022-12-01T10:31:02.1421771Z   test_send_recv_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28087
2022-12-01T10:31:02.1421930Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28088
2022-12-01T10:31:02.1422083Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 28089
2022-12-01T10:31:02.1422359Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.1422482Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.1422766Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.1422904Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.1423082Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:31:02.1423340Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.1423463Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.1423746Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.1423881Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.1424059Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:31:02.1424386Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.1424542Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.1424826Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.1424961Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.1425127Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:31:02.1425428Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.1425725Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.1426021Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.1426194Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:31:02.1426362Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:31:02.1426526Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:31:02.1426778Z STAGE:2022-12-01 10:30:28 28089:28089 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.1427020Z STAGE:2022-12-01 10:30:28 28087:28087 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.1427245Z STAGE:2022-12-01 10:30:28 28088:28088 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.1427501Z STAGE:2022-12-01 10:30:28 28089:28089 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.1427748Z STAGE:2022-12-01 10:30:28 28087:28087 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.1427992Z STAGE:2022-12-01 10:30:28 28088:28088 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.1428063Z ok (1.755s)
2022-12-01T10:31:02.1428068Z 
2022-12-01T10:31:02.1428266Z ----------------------------------------------------------------------
2022-12-01T10:31:02.1428345Z Ran 1 test in 1.755s
2022-12-01T10:31:02.1428350Z 
2022-12-01T10:31:02.1428409Z OK
2022-12-01T10:31:02.1428413Z 
2022-12-01T10:31:02.1428497Z Generating XML reports...
2022-12-01T10:31:02.1428832Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103027.xml
2022-12-01T10:31:02.1429067Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.1429343Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.1429467Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.1429751Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.1429890Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.1429895Z 
2022-12-01T10:31:02.1429968Z Running tests...
2022-12-01T10:31:02.1430165Z ----------------------------------------------------------------------
2022-12-01T10:31:02.1430386Z   test_send_recv_with_tag (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28143
2022-12-01T10:31:02.1430543Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28144
2022-12-01T10:31:02.1430697Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 28145
2022-12-01T10:31:02.1430968Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.1431092Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.1431413Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.1431576Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.1431756Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:31:02.1432030Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.1432144Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.1432422Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.1432561Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.1432739Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:31:02.1433017Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.1433143Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.1433422Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.1433559Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.1433724Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:31:02.1434030Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.1434329Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.1434623Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.1434791Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:31:02.1434960Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:31:02.1435123Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:31:02.1435190Z ok (1.852s)
2022-12-01T10:31:02.1435195Z 
2022-12-01T10:31:02.1435390Z ----------------------------------------------------------------------
2022-12-01T10:31:02.1435453Z Ran 1 test in 1.853s
2022-12-01T10:31:02.1435471Z 
2022-12-01T10:31:02.1435519Z OK
2022-12-01T10:31:02.1435524Z 
2022-12-01T10:31:02.1435607Z Generating XML reports...
2022-12-01T10:31:02.1435949Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103030.xml
2022-12-01T10:31:02.1436222Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.1436580Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.1436706Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.1436990Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.1437129Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.1437134Z 
2022-12-01T10:31:02.1437194Z Running tests...
2022-12-01T10:31:02.1437388Z ----------------------------------------------------------------------
2022-12-01T10:31:02.1437644Z   test_send_recv_with_tag_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28196
2022-12-01T10:31:02.1437805Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28197
2022-12-01T10:31:02.1437961Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 28198
2022-12-01T10:31:02.1438275Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.1438429Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.1438711Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.1438834Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.1439013Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:31:02.1439286Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.1439409Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.1439689Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.1439827Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.1440008Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:31:02.1440283Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.1440406Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.1440674Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.1440812Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.1440985Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:31:02.1441286Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.1441587Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.1441880Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.1442050Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:31:02.1442211Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:31:02.1442374Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:31:02.1442612Z STAGE:2022-12-01 10:30:35 28196:28196 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.1442854Z STAGE:2022-12-01 10:30:35 28197:28197 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.1443094Z STAGE:2022-12-01 10:30:35 28198:28198 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.1443346Z STAGE:2022-12-01 10:30:35 28196:28196 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.1443594Z STAGE:2022-12-01 10:30:35 28197:28197 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.1443832Z STAGE:2022-12-01 10:30:35 28198:28198 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.1443902Z ok (1.856s)
2022-12-01T10:31:02.1443907Z 
2022-12-01T10:31:02.1444107Z ----------------------------------------------------------------------
2022-12-01T10:31:02.1444188Z Ran 1 test in 1.856s
2022-12-01T10:31:02.1444193Z 
2022-12-01T10:31:02.1444242Z OK
2022-12-01T10:31:02.1444246Z 
2022-12-01T10:31:02.1444332Z Generating XML reports...
2022-12-01T10:31:02.1444678Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103033.xml
2022-12-01T10:31:02.1444911Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.1445224Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.1445387Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.1445675Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.1445812Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.1445817Z 
2022-12-01T10:31:02.1445878Z Running tests...
2022-12-01T10:31:02.1446075Z ----------------------------------------------------------------------
2022-12-01T10:31:02.1446325Z   test_send_recv_with_tag_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28252
2022-12-01T10:31:02.1446484Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28253
2022-12-01T10:31:02.1446643Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 28254
2022-12-01T10:31:02.1446973Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.1447147Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.1447471Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.1447611Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.1447838Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:31:02.1448116Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.1448239Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.1448524Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.1448661Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.1448844Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:31:02.1449119Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.1449243Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.1449508Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.1449642Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.1449818Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:31:02.1450120Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.1450421Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.1450716Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.1450880Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:31:02.1451128Z STAGE:2022-12-01 10:30:38 28254:28254 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.1451289Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:31:02.1451439Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:31:02.1451683Z STAGE:2022-12-01 10:30:38 28252:28252 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.1451961Z STAGE:2022-12-01 10:30:38 28253:28253 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:31:02.1452210Z STAGE:2022-12-01 10:30:38 28252:28252 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.1452481Z STAGE:2022-12-01 10:30:38 28253:28253 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.1452720Z STAGE:2022-12-01 10:30:38 28254:28254 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:31:02.1452789Z ok (1.853s)
2022-12-01T10:31:02.1452794Z 
2022-12-01T10:31:02.1452991Z ----------------------------------------------------------------------
2022-12-01T10:31:02.1453065Z Ran 1 test in 1.853s
2022-12-01T10:31:02.1453070Z 
2022-12-01T10:31:02.1453118Z OK
2022-12-01T10:31:02.1453123Z 
2022-12-01T10:31:02.1453208Z Generating XML reports...
2022-12-01T10:31:02.1453554Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103037.xml
2022-12-01T10:31:02.1453791Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.1454069Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.1454197Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.1454480Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.1454617Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.1454622Z 
2022-12-01T10:31:02.1454695Z Running tests...
2022-12-01T10:31:02.1454878Z ----------------------------------------------------------------------
2022-12-01T10:31:02.1455115Z   test_sparse_all_reduce_sum (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28308
2022-12-01T10:31:02.1455383Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28309
2022-12-01T10:31:02.1455546Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 28310
2022-12-01T10:31:02.1455824Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.1455947Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.1456231Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.1456367Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.1456530Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:31:02.1456801Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.1456922Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.1457204Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.1457341Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.1457518Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:31:02.1457824Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.1458096Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.1458221Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.1458487Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.1458622Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.1458830Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:31:02.1459134Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.1459462Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.1459627Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:31:02.1459791Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:31:02.1459953Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:31:02.1460020Z ok (1.953s)
2022-12-01T10:31:02.1460025Z 
2022-12-01T10:31:02.1460208Z ----------------------------------------------------------------------
2022-12-01T10:31:02.1460282Z Ran 1 test in 1.953s
2022-12-01T10:31:02.1460287Z 
2022-12-01T10:31:02.1460436Z OK
2022-12-01T10:31:02.1460441Z 
2022-12-01T10:31:02.1460529Z Generating XML reports...
2022-12-01T10:31:02.1460873Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103040.xml
2022-12-01T10:31:02.1461109Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.1461389Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.1461514Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.1461782Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.1461918Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.1461923Z 
2022-12-01T10:31:02.1461999Z Running tests...
2022-12-01T10:31:02.1462196Z ----------------------------------------------------------------------
2022-12-01T10:31:02.1462445Z   test_sparse_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28388
2022-12-01T10:31:02.1462605Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28389
2022-12-01T10:31:02.1462760Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 28390
2022-12-01T10:31:02.1463035Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.1463160Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.1463428Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.1463569Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.1463749Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:31:02.1464021Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.1464146Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.1464423Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.1464556Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.1464734Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:31:02.1464992Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.1465113Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.1465392Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.1465578Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.1465757Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:31:02.1466094Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.1466389Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.1466680Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.1466847Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:31:02.1466998Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:31:02.1467159Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:31:02.1467258Z skip: CUDA is not available. (1.751s)
2022-12-01T10:31:02.1467264Z 
2022-12-01T10:31:02.1467460Z ----------------------------------------------------------------------
2022-12-01T10:31:02.1467537Z Ran 1 test in 1.751s
2022-12-01T10:31:02.1467542Z 
2022-12-01T10:31:02.1467616Z OK (skipped=1)
2022-12-01T10:31:02.1467622Z 
2022-12-01T10:31:02.1467706Z Generating XML reports...
2022-12-01T10:31:02.1468049Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103043.xml
2022-12-01T10:31:02.1468282Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.1468544Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.1468668Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.1468948Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.1469085Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.1469093Z 
2022-12-01T10:31:02.1469168Z Running tests...
2022-12-01T10:31:02.1469364Z ----------------------------------------------------------------------
2022-12-01T10:31:02.1469602Z   test_stateless_api_with_ddp (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28441
2022-12-01T10:31:02.1469758Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28442
2022-12-01T10:31:02.1469909Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 28443
2022-12-01T10:31:02.1470167Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.1470289Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.1470571Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.1470707Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.1470887Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:31:02.1471157Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.1471281Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.1471562Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.1471683Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.1471861Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:31:02.1472158Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.1472458Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.1472609Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.1472888Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.1473022Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.1473196Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:31:02.1473494Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.1473778Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.1473944Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:31:02.1474107Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:31:02.1474270Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:31:02.1474376Z skip: Need at least 2 CUDA devices (1.753s)
2022-12-01T10:31:02.1474382Z 
2022-12-01T10:31:02.1474576Z ----------------------------------------------------------------------
2022-12-01T10:31:02.1474651Z Ran 1 test in 1.753s
2022-12-01T10:31:02.1474657Z 
2022-12-01T10:31:02.1474727Z OK (skipped=1)
2022-12-01T10:31:02.1474732Z 
2022-12-01T10:31:02.1474816Z Generating XML reports...
2022-12-01T10:31:02.1475146Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103047.xml
2022-12-01T10:31:02.1475378Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.1475653Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.1475780Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.1476059Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.1476192Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.1476198Z 
2022-12-01T10:31:02.1476270Z Running tests...
2022-12-01T10:31:02.1476467Z ----------------------------------------------------------------------
2022-12-01T10:31:02.1476688Z   test_static_graph_api_cpu (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28494
2022-12-01T10:31:02.1476844Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28495
2022-12-01T10:31:02.1476999Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 28496
2022-12-01T10:31:02.1477276Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.1477406Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.1477683Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.1477821Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.1477999Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:31:02.1478270Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.1478380Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.1478658Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.1478822Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.1479002Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:31:02.1479297Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.1479418Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.1479695Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.1479828Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.1480003Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:31:02.1480292Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.1480590Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.1480885Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.1481052Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:31:02.1481212Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:31:02.1481397Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp94av6d9u
2022-12-01T10:31:02.1481582Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6m566qj2
2022-12-01T10:31:02.1481776Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp94av6d9u/_remote_module_non_scriptable.py
2022-12-01T10:31:02.1481967Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6m566qj2/_remote_module_non_scriptable.py
2022-12-01T10:31:02.1482123Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:31:02.1482309Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpeahjf9r7
2022-12-01T10:31:02.1482503Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpeahjf9r7/_remote_module_non_scriptable.py
2022-12-01T10:31:02.1482570Z ok (1.852s)
2022-12-01T10:31:02.1482575Z 
2022-12-01T10:31:02.1482771Z ----------------------------------------------------------------------
2022-12-01T10:31:02.1482846Z Ran 1 test in 1.853s
2022-12-01T10:31:02.1482851Z 
2022-12-01T10:31:02.1482910Z OK
2022-12-01T10:31:02.1482916Z 
2022-12-01T10:31:02.1482998Z Generating XML reports...
2022-12-01T10:31:02.1483326Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103050.xml
2022-12-01T10:31:02.1483561Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.1483835Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.1483959Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.1484237Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.1484370Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.1484375Z 
2022-12-01T10:31:02.1484450Z Running tests...
2022-12-01T10:31:02.1484642Z ----------------------------------------------------------------------
2022-12-01T10:31:02.1484871Z   test_sync_bn_logged (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28547
2022-12-01T10:31:02.1485015Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28548
2022-12-01T10:31:02.1485170Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 28549
2022-12-01T10:31:02.1485471Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.1485622Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.1485903Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.1486036Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.1486214Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:31:02.1486486Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.1486597Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.1486877Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.1487014Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.1487192Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:31:02.1487465Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.1487587Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.1487865Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.1487999Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.1488174Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:31:02.1488464Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.1488764Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.1489059Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.1489223Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:31:02.1489386Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:31:02.1489543Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:31:02.1489647Z skip: Need at least 2 CUDA devices (1.857s)
2022-12-01T10:31:02.1489652Z 
2022-12-01T10:31:02.1489848Z ----------------------------------------------------------------------
2022-12-01T10:31:02.1489922Z Ran 1 test in 1.857s
2022-12-01T10:31:02.1489927Z 
2022-12-01T10:31:02.1489987Z OK (skipped=1)
2022-12-01T10:31:02.1489992Z 
2022-12-01T10:31:02.1490075Z Generating XML reports...
2022-12-01T10:31:02.1490414Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103053.xml
2022-12-01T10:31:02.1490646Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.1490916Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.1491038Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.1491317Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.1491450Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.1491455Z 
2022-12-01T10:31:02.1491515Z Running tests...
2022-12-01T10:31:02.1491708Z ----------------------------------------------------------------------
2022-12-01T10:31:02.1491994Z   test_undefined_grad_parity_unused_parameters (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28600
2022-12-01T10:31:02.1492187Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28601
2022-12-01T10:31:02.1492340Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 28602
2022-12-01T10:31:02.1492612Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.1492736Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.1493019Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.1493154Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.1493318Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:31:02.1493590Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.1493712Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.1493990Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.1494125Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.1494301Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:31:02.1494573Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.1494695Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.1494979Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.1495102Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.1495368Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:31:02.1495675Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.1495971Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.1496260Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:31:02.1496425Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:31:02.1496590Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:31:02.1496749Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:31:02.1496855Z skip: Need at least 2 CUDA devices (1.854s)
2022-12-01T10:31:02.1496861Z 
2022-12-01T10:31:02.1497046Z ----------------------------------------------------------------------
2022-12-01T10:31:02.1497124Z Ran 1 test in 1.854s
2022-12-01T10:31:02.1497129Z 
2022-12-01T10:31:02.1497203Z OK (skipped=1)
2022-12-01T10:31:02.1497207Z 
2022-12-01T10:31:02.1497292Z Generating XML reports...
2022-12-01T10:31:02.1497633Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103056.xml
2022-12-01T10:31:02.1497871Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.1498145Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.1498267Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.1498533Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.1498707Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.1498713Z 
2022-12-01T10:31:02.1498813Z Running tests...
2022-12-01T10:31:02.1499007Z ----------------------------------------------------------------------
2022-12-01T10:31:02.1499341Z   test_verify_model_across_rank_with_logger (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'nccl', 'gloo'} (0.001s)
2022-12-01T10:31:02.1499347Z 
2022-12-01T10:31:02.1499540Z ----------------------------------------------------------------------
2022-12-01T10:31:02.1499615Z Ran 1 test in 0.002s
2022-12-01T10:31:02.1499620Z 
2022-12-01T10:31:02.1499690Z OK (skipped=1)
2022-12-01T10:31:02.1499695Z 
2022-12-01T10:31:02.1499777Z Generating XML reports...
2022-12-01T10:31:02.1500103Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103100.xml
2022-12-01T10:31:02.1500334Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:31:02.1500749Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:31:02.1500876Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:31:02.1501157Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:31:02.1501294Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:31:02.1501299Z 
2022-12-01T10:31:02.1501372Z Running tests...
2022-12-01T10:31:02.1501568Z ----------------------------------------------------------------------
2022-12-01T10:31:02.1501905Z   test_verify_model_across_rank_without_logger (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'nccl', 'gloo'} (0.001s)
2022-12-01T10:31:02.1501911Z 
2022-12-01T10:31:02.1502092Z ----------------------------------------------------------------------
2022-12-01T10:31:02.1502167Z Ran 1 test in 0.002s
2022-12-01T10:31:02.1502174Z 
2022-12-01T10:31:02.1502245Z OK (skipped=1)
2022-12-01T10:31:02.1502251Z 
2022-12-01T10:31:02.1502335Z Generating XML reports...
2022-12-01T10:31:02.1502677Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103101.xml
2022-12-01T10:31:02.1502682Z 
2022-12-01T10:31:02.1503065Z ##[endgroup]
2022-12-01T10:31:02.1503423Z FINISHED PRINTING LOG FILE of distributed/test_distributed_spawn (/var/lib/jenkins/workspace/test/test-reports/distributed-test_distributed_spawn_lt5tdku2)
2022-12-01T10:31:02.1503429Z 
2022-12-01T10:31:02.1503593Z Running distributed tests for the test backend with file init_method
2022-12-01T10:31:02.1503728Z Running distributed tests for the gloo backend with file init_method
2022-12-01T10:31:02.1504108Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-12-01 10:31:01.948555]
2022-12-01T10:31:02.1504501Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-12-01 10:31:01.949271]
2022-12-01T10:31:03.3342436Z 
2022-12-01T10:31:03.3342916Z Expand the folded group to see the log file of distributed/test_distributed_spawn
2022-12-01T10:31:03.3343649Z ##[group]PRINTING LOG FILE of distributed/test_distributed_spawn (/var/lib/jenkins/workspace/test/test-reports/distributed-test_distributed_spawn_lilki7ve)
2022-12-01T10:31:03.3344023Z <unittest.suite.TestSuite tests=[]>
2022-12-01T10:31:03.3344170Z 
2022-12-01T10:31:03.3344384Z ##[endgroup]
2022-12-01T10:31:03.3344930Z FINISHED PRINTING LOG FILE of distributed/test_distributed_spawn (/var/lib/jenkins/workspace/test/test-reports/distributed-test_distributed_spawn_lilki7ve)
2022-12-01T10:31:03.3345189Z 
2022-12-01T10:43:19.1273009Z 
2022-12-01T10:43:19.1273462Z Expand the folded group to see the log file of distributed/test_distributed_spawn
2022-12-01T10:43:19.1274451Z ##[group]PRINTING LOG FILE of distributed/test_distributed_spawn (/var/lib/jenkins/workspace/test/test-reports/distributed-test_distributed_spawn_z513gh_5)
2022-12-01T10:43:19.1285369Z <unittest.suite.TestSuite tests=[]>
2022-12-01T10:43:19.1325398Z <unittest.suite.TestSuite tests=[<__main__.TestDistBackendWithSpawn testMethod=test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager>, <__main__.TestDistBackendWithSpawn testMethod=test_3_level_hierarchical_model_averager>, <__main__.TestDistBackendWithSpawn testMethod=test_Backend_enum_class>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallelCPU>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallelCPU_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_2D_Input>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Channels_Last>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_No_Affine>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_non_default_stream>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_requires_grad>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_with_amp_and_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedSampler_padding>, <__main__.TestDistBackendWithSpawn testMethod=test_SyncBatchNorm_process_group>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync_allreduce_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync_allreduce_with_then_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_simple>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_with_empty>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_into_cat_tensor_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_into_stack_tensor_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_multigpu_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_object_default_pg>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_object_subgroup>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_v_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_max_complex_unsupported>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_complex_unsupported_ops>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_multigpu_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_result_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_async>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_cuda_async>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_average_parameters>, <__main__.TestDistBackendWithSpawn testMethod=test_backend_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_backend_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_timeout_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_timeout_global>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_timeout_group>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_gloo>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_gloo_tags>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_mixed_backend_err>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_no_rank_zero_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_op_err>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_op_list_err>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_ring_exchange_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_self_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_tensor_err>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_group>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_object_list>, <__main__.TestDistBackendWithSpawn testMethod=test_compute_bucket_assignment_by_size_sparse_error_with_logger>, <__main__.TestDistBackendWithSpawn testMethod=test_compute_bucket_assignment_by_size_sparse_error_without_logger>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_broadcast_buffer>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_broadcast_buffer_via_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_buffer_hook_allreduce>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_buffer_hook_allreduce_return_future>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_build_debug_param_to_name_mapping>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_build_debug_param_to_name_mapping_requires_grad>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_comm_hook_logging>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_control_flow_different_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_control_flow_same_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_create_graph>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_device>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_forward_backward_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_grad_div_uneven_inputs>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_allreduce>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_allreduce_process_group>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_post_localSGD>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_powerSGD>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_pickling_powerSGD>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adam_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adam_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_ignore_params_arg>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_inference>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_join_model_equivalence>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_logging_data_cpu>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_logging_data_gpu>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_model_diff_num_params_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_model_diff_shape_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_multiple_nested_unused_params_err_ignore_params>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_multiple_nested_unused_params_error>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_namedtuple>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_new_tensor_in_fwd>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_new_tensor_in_fwd_static_graph>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_profiling_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_profiling_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_python_error_logged>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_returns_tensor_with_no_grad>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_shared_grad_acc_unused_params>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_static_graph_nested_types>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_sync_bn_training_vs_eval>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_sync_module_states>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_input_exception>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_input_join_disable>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_inputs>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_inputs_stop_iteration_sync_bn>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_unused_params_rebuild_buckets_exception>, <__main__.TestDistBackendWithSpawn testMethod=test_destroy_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_destroy_group>, <__main__.TestDistBackendWithSpawn testMethod=test_detect_ddp_is_actually_static>, <__main__.TestDistBackendWithSpawn testMethod=test_different_graph_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_dump_DDP_relevant_env_vars>, <__main__.TestDistBackendWithSpawn testMethod=test_gather>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_checks>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_group>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_object>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_object_subgroup>, <__main__.TestDistBackendWithSpawn testMethod=test_get_backend>, <__main__.TestDistBackendWithSpawn testMethod=test_get_future>, <__main__.TestDistBackendWithSpawn testMethod=test_get_rank>, <__main__.TestDistBackendWithSpawn testMethod=test_get_rank_size_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_get_rank_size_group>, <__main__.TestDistBackendWithSpawn testMethod=test_invalid_static_graph>, <__main__.TestDistBackendWithSpawn testMethod=test_irecv>, <__main__.TestDistBackendWithSpawn testMethod=test_isend>, <__main__.TestDistBackendWithSpawn testMethod=test_isend_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_isend_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_allreduce_hang>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_allreduce_hang_wait_all_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_failure_order>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_gloo>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_gloo_rank_0_timeout>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_gloo_subgroup>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_wait_all_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_allgather>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_allreduce>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_broadcast>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_reduce>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_high_priority_stream>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_by_enumeration>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_by_enumeration_input_rank_exceeds_world_size>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_by_enumeration_negative_input_rank>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_group_size_exceeds_world_size>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_overlap_not_allowed>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_world_size_not_divisible_by_group_size>, <__main__.TestDistBackendWithSpawn testMethod=test_output_unused_in_loss_dict_module>, <__main__.TestDistBackendWithSpawn testMethod=test_output_unused_in_loss_tuple_module>, <__main__.TestDistBackendWithSpawn testMethod=test_periodic_model_averager>, <__main__.TestDistBackendWithSpawn testMethod=test_periodic_model_averager_param_group>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity_with_hierarchical_sgd>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_step_reload>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_max>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_min>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_product>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_scatter_tensor_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_scatter_v_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum_cuda_twice>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum_twice>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_checks>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_group>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_object_list>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_any_source>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_any_source_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_any_source_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_nccl_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_nccl_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_with_tag>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_with_tag_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_with_tag_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_sparse_all_reduce_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_sparse_all_reduce_sum_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_stateless_api_with_ddp>, <__main__.TestDistBackendWithSpawn testMethod=test_static_graph_api_cpu>, <__main__.TestDistBackendWithSpawn testMethod=test_sync_bn_logged>, <__main__.TestDistBackendWithSpawn testMethod=test_undefined_grad_parity_unused_parameters>, <__main__.TestDistBackendWithSpawn testMethod=test_verify_model_across_rank_with_logger>, <__main__.TestDistBackendWithSpawn testMethod=test_verify_model_across_rank_without_logger>]>
2022-12-01T10:43:19.1363914Z test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1365923Z test_3_level_hierarchical_model_averager (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1366444Z test_Backend_enum_class (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1366955Z test_DistributedDataParallel (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1367522Z test_DistributedDataParallelCPU (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1368107Z test_DistributedDataParallelCPU_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1368713Z test_DistributedDataParallel_SyncBatchNorm (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1369284Z test_DistributedDataParallel_SyncBatchNorm_2D_Input (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1370074Z test_DistributedDataParallel_SyncBatchNorm_Channels_Last (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1370805Z test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1403178Z test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1403932Z test_DistributedDataParallel_SyncBatchNorm_No_Affine (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1404424Z test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1404870Z test_DistributedDataParallel_non_default_stream (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1405338Z test_DistributedDataParallel_requires_grad (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1406026Z test_DistributedDataParallel_with_amp_and_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1406421Z test_DistributedSampler_padding (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1406866Z test_SyncBatchNorm_process_group (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1407419Z test_accumulate_gradients_no_sync (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1407992Z test_accumulate_gradients_no_sync_allreduce_hook (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1408433Z test_accumulate_gradients_no_sync_allreduce_with_then_hook (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1408789Z test_accumulate_gradients_no_sync_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1409114Z test_all_gather (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1409411Z test_all_gather_coalesced_complex (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1409716Z test_all_gather_coalesced_full_group (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1410030Z test_all_gather_coalesced_group (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1410349Z test_all_gather_coalesced_simple (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1410652Z test_all_gather_coalesced_with_empty (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1410960Z test_all_gather_complex (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1411247Z test_all_gather_cuda (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1411540Z test_all_gather_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1411825Z test_all_gather_full_group (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1412111Z test_all_gather_group (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1412412Z test_all_gather_into_cat_tensor_cuda (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1412712Z test_all_gather_into_stack_tensor_cuda (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1413015Z test_all_gather_multigpu (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1413390Z test_all_gather_multigpu_complex (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1413695Z test_all_gather_object_default_pg (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1414004Z test_all_gather_object_subgroup (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1414304Z test_all_gather_v_cuda (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1414612Z test_all_reduce_coalesced_full_group_max (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1414929Z test_all_reduce_coalesced_full_group_min (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1415264Z test_all_reduce_coalesced_full_group_product (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1415598Z test_all_reduce_coalesced_full_group_sum (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1415903Z test_all_reduce_coalesced_group_max (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1416218Z test_all_reduce_coalesced_group_min (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1416536Z test_all_reduce_coalesced_group_product (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1416984Z test_all_reduce_coalesced_group_sum (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1417280Z test_all_reduce_coalesced_max (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1417662Z test_all_reduce_coalesced_max_complex_unsupported (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1418027Z test_all_reduce_coalesced_min (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1418320Z test_all_reduce_coalesced_product (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1418631Z test_all_reduce_coalesced_sum (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1418947Z test_all_reduce_complex_unsupported_ops (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1419250Z test_all_reduce_full_group_max (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1419552Z test_all_reduce_full_group_min (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1419863Z test_all_reduce_full_group_product (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1420170Z test_all_reduce_full_group_sum (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1420622Z test_all_reduce_group_max (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1420925Z test_all_reduce_group_min (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1421222Z test_all_reduce_group_product (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1421504Z test_all_reduce_group_sum (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1421787Z test_all_reduce_max (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1422063Z test_all_reduce_min (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1422336Z test_all_reduce_multigpu (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1428181Z test_all_reduce_multigpu_complex (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1428665Z test_all_reduce_product (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1430325Z test_all_reduce_result_cuda (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1430787Z test_all_reduce_sum (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1431269Z test_all_reduce_sum_async (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1431816Z test_all_reduce_sum_complex (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1432261Z test_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1432562Z test_all_reduce_sum_cuda_async (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1432856Z test_all_reduce_sum_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1433144Z test_all_to_all (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1433430Z test_all_to_all_complex (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1433709Z test_all_to_all_cuda (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1434024Z test_all_to_all_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1434321Z test_all_to_all_full_group (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1434675Z test_all_to_all_full_group_cuda (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1435177Z test_all_to_all_group (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1435640Z test_all_to_all_group_cuda (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1436149Z test_all_to_all_single_equal_split (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1436718Z test_all_to_all_single_equal_split_complex (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1437063Z test_all_to_all_single_equal_split_cuda (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1437400Z test_all_to_all_single_equal_split_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1437726Z test_all_to_all_single_equal_split_full_group (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1438067Z test_all_to_all_single_equal_split_full_group_cuda (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1438395Z test_all_to_all_single_equal_split_group (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1438728Z test_all_to_all_single_equal_split_group_cuda (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1439331Z test_all_to_all_single_unequal_split (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1439945Z test_all_to_all_single_unequal_split_complex (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1440449Z test_all_to_all_single_unequal_split_cuda (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1470645Z test_all_to_all_single_unequal_split_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1471307Z test_all_to_all_single_unequal_split_full_group (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1471962Z test_all_to_all_single_unequal_split_full_group_cuda (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1472530Z test_all_to_all_single_unequal_split_group (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1592912Z test_all_to_all_single_unequal_split_group_cuda (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1593296Z test_average_parameters (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1593606Z test_backend_full_group (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1593876Z test_backend_group (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1594157Z test_barrier (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1594429Z test_barrier_cuda (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1594713Z test_barrier_full_group (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1594989Z test_barrier_full_group_cuda (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1595279Z test_barrier_group (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1595563Z test_barrier_group_cuda (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1595849Z test_barrier_timeout_full_group (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1596150Z test_barrier_timeout_global (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1596445Z test_barrier_timeout_group (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1596723Z test_batch_isend_irecv_gloo (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1597023Z test_batch_isend_irecv_gloo_tags (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1597342Z test_batch_isend_irecv_mixed_backend_err (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1597651Z test_batch_isend_irecv_nccl (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1597942Z test_batch_isend_irecv_no_rank_zero_nccl (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1603236Z test_batch_isend_irecv_op_err (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1603553Z test_batch_isend_irecv_op_list_err (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1603859Z test_batch_isend_irecv_ring_exchange_nccl (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1604179Z test_batch_isend_irecv_self_nccl (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1604487Z test_batch_isend_irecv_tensor_err (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1604773Z test_broadcast (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1605036Z test_broadcast_cuda (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1605330Z test_broadcast_full_group (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1605626Z test_broadcast_group (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1605899Z test_broadcast_multigpu (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1606190Z test_broadcast_object_list (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1606529Z test_compute_bucket_assignment_by_size_sparse_error_with_logger (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1606900Z test_compute_bucket_assignment_by_size_sparse_error_without_logger (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1607239Z test_ddp_broadcast_buffer (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1607544Z test_ddp_broadcast_buffer_via_hook (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1607852Z test_ddp_buffer_hook_allreduce (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1608156Z test_ddp_buffer_hook_allreduce_return_future (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1608653Z test_ddp_build_debug_param_to_name_mapping (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1609071Z test_ddp_build_debug_param_to_name_mapping_requires_grad (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1609388Z test_ddp_comm_hook_logging (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1609703Z test_ddp_control_flow_different_across_ranks (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1610034Z test_ddp_control_flow_same_across_ranks (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1610342Z test_ddp_create_graph (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1610604Z test_ddp_device (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1610894Z test_ddp_forward_backward_hook (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1611202Z test_ddp_grad_div_uneven_inputs (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1611493Z test_ddp_hook_parity_allreduce (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1611821Z test_ddp_hook_parity_allreduce_process_group (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1612150Z test_ddp_hook_parity_post_localSGD (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1612445Z test_ddp_hook_parity_powerSGD (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1612751Z test_ddp_hook_pickling_powerSGD (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1613093Z test_ddp_hook_with_optimizer_parity_adam_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1613562Z test_ddp_hook_with_optimizer_parity_adam_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1613974Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1614437Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1614905Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1615364Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1615804Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1616261Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1616714Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1617162Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1617554Z test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1617925Z test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1618252Z test_ddp_ignore_params_arg (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1618537Z test_ddp_inference (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1618817Z test_ddp_join_model_equivalence (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1619117Z test_ddp_logging_data_cpu (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1619409Z test_ddp_logging_data_gpu (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1619708Z test_ddp_model_diff_num_params_across_ranks (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1620031Z test_ddp_model_diff_shape_across_ranks (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1620547Z test_ddp_multiple_nested_unused_params_err_ignore_params (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1620967Z test_ddp_multiple_nested_unused_params_error (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1621305Z test_ddp_namedtuple (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1621594Z test_ddp_new_tensor_in_fwd (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1621901Z test_ddp_new_tensor_in_fwd_static_graph (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1622213Z test_ddp_profiling_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1622536Z test_ddp_profiling_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1622847Z test_ddp_python_error_logged (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1623139Z test_ddp_returns_tensor_with_no_grad (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1623462Z test_ddp_shared_grad_acc_unused_params (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1623777Z test_ddp_static_graph_nested_types (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1624090Z test_ddp_sync_bn_training_vs_eval (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1624378Z test_ddp_sync_module_states (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1624677Z test_ddp_uneven_input_exception (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1624987Z test_ddp_uneven_input_join_disable (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1625273Z test_ddp_uneven_inputs (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1625583Z test_ddp_uneven_inputs_stop_iteration_sync_bn (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1625923Z test_ddp_unused_params_rebuild_buckets_exception (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1626239Z test_destroy_full_group (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1626508Z test_destroy_group (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1626801Z test_detect_ddp_is_actually_static (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1627113Z test_different_graph_across_ranks (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1627407Z test_dump_DDP_relevant_env_vars (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1627689Z test_gather (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1627956Z test_gather_checks (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1628216Z test_gather_cuda (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1628494Z test_gather_full_group (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1628771Z test_gather_group (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1629032Z test_gather_object (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1629314Z test_gather_object_subgroup (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1629595Z test_get_backend (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1629860Z test_get_future (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1630109Z test_get_rank (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1630392Z test_get_rank_size_full_group (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1630685Z test_get_rank_size_group (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1630961Z test_invalid_static_graph (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1631233Z test_irecv (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1631489Z test_isend (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1631755Z test_isend_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1632050Z test_isend_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1632353Z test_monitored_barrier_allreduce_hang (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1632691Z test_monitored_barrier_allreduce_hang_wait_all_ranks (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1633011Z test_monitored_barrier_failure_order (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1633318Z test_monitored_barrier_gloo (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1633662Z test_monitored_barrier_gloo_rank_0_timeout (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1633972Z test_monitored_barrier_gloo_subgroup (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1634318Z test_monitored_barrier_wait_all_ranks (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1634630Z test_nccl_backend_bool_allgather (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1634937Z test_nccl_backend_bool_allreduce (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1635230Z test_nccl_backend_bool_broadcast (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1635531Z test_nccl_backend_bool_reduce (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1635834Z test_nccl_high_priority_stream (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1636111Z test_new_subgroups (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1636404Z test_new_subgroups_by_enumeration (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1636748Z test_new_subgroups_by_enumeration_input_rank_exceeds_world_size (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1637102Z test_new_subgroups_by_enumeration_negative_input_rank (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1637452Z test_new_subgroups_group_size_exceeds_world_size (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1637781Z test_new_subgroups_overlap_not_allowed (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1638120Z test_new_subgroups_world_size_not_divisible_by_group_size (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1638441Z test_output_unused_in_loss_dict_module (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1638760Z test_output_unused_in_loss_tuple_module (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1639070Z test_periodic_model_averager (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1639374Z test_periodic_model_averager_param_group (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1639698Z test_post_localSGD_optimizer_parity (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1640031Z test_post_localSGD_optimizer_parity_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1640389Z test_post_localSGD_optimizer_parity_with_hierarchical_sgd (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1640752Z test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1641109Z test_post_localSGD_optimizer_step_reload (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1641419Z test_reduce_full_group_max (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1641697Z test_reduce_full_group_min (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1641996Z test_reduce_full_group_product (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1642293Z test_reduce_full_group_sum (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1642579Z test_reduce_group_max (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1642845Z test_reduce_group_min (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1643131Z test_reduce_group_product (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1643417Z test_reduce_group_sum (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1643680Z test_reduce_max (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1643947Z test_reduce_min (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1644222Z test_reduce_multigpu (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1644505Z test_reduce_product (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1644798Z test_reduce_scatter_tensor_cuda (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1645099Z test_reduce_scatter_v_cuda (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1645373Z test_reduce_sum (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1645632Z test_reduce_sum_cuda (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1645915Z test_reduce_sum_cuda_twice (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1646201Z test_reduce_sum_twice (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1646525Z test_scatter (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1646798Z test_scatter_checks (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1647112Z test_scatter_complex (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1647375Z test_scatter_cuda (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1647660Z test_scatter_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1647953Z test_scatter_full_group (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1648234Z test_scatter_group (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1648505Z test_scatter_object_list (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1648783Z test_send_recv (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1649062Z test_send_recv_any_source (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1649364Z test_send_recv_any_source_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1649693Z test_send_recv_any_source_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1650012Z test_send_recv_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1650294Z test_send_recv_nccl (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1650596Z test_send_recv_nccl_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1650908Z test_send_recv_nccl_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1651212Z test_send_recv_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1651492Z test_send_recv_with_tag (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1651795Z test_send_recv_with_tag_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1652116Z test_send_recv_with_tag_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1652409Z test_sparse_all_reduce_sum (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1652705Z test_sparse_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1653006Z test_stateless_api_with_ddp (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1653370Z test_static_graph_api_cpu (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1653665Z test_sync_bn_logged (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1653975Z test_undefined_grad_parity_unused_parameters (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1654310Z test_verify_model_across_rank_with_logger (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1654628Z test_verify_model_across_rank_without_logger (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1655146Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.1655631Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.1655956Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.1656396Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.1656750Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.1656922Z 
2022-12-01T10:43:19.1656995Z Running tests...
2022-12-01T10:43:19.1657289Z ----------------------------------------------------------------------
2022-12-01T10:43:19.1657749Z   test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28698
2022-12-01T10:43:19.1658205Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28699
2022-12-01T10:43:19.1658538Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 28700
2022-12-01T10:43:19.1658975Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.1659305Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.1659777Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.1660152Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.1660667Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:43:19.1661156Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.1661487Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.1661903Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.1662252Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.1662591Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:43:19.1663052Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.1663380Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.1663808Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.1664151Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.1664472Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:43:19.1664970Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.1665494Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.1666009Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.1666380Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:43:19.1666728Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:43:19.1667072Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:43:19.1667346Z skip: Need at least 2 CUDA devices (1.853s)
2022-12-01T10:43:19.1667484Z 
2022-12-01T10:43:19.1667684Z ----------------------------------------------------------------------
2022-12-01T10:43:19.1667924Z Ran 1 test in 1.854s
2022-12-01T10:43:19.1668037Z 
2022-12-01T10:43:19.1668108Z OK (skipped=1)
2022-12-01T10:43:19.1668219Z 
2022-12-01T10:43:19.1668291Z Generating XML reports...
2022-12-01T10:43:19.1668746Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103104.xml
2022-12-01T10:43:19.1669242Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.1669706Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.1670027Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.1670459Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.1670803Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.1670968Z 
2022-12-01T10:43:19.1671027Z Running tests...
2022-12-01T10:43:19.1671328Z ----------------------------------------------------------------------
2022-12-01T10:43:19.1671698Z   test_3_level_hierarchical_model_averager (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.003s)
2022-12-01T10:43:19.1671922Z 
2022-12-01T10:43:19.1672118Z ----------------------------------------------------------------------
2022-12-01T10:43:19.1672343Z Ran 1 test in 0.003s
2022-12-01T10:43:19.1672538Z 
2022-12-01T10:43:19.1672613Z OK (skipped=1)
2022-12-01T10:43:19.1672767Z 
2022-12-01T10:43:19.1672852Z Generating XML reports...
2022-12-01T10:43:19.1673299Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103107.xml
2022-12-01T10:43:19.1673796Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.1674260Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.1674595Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.1675011Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.1675365Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.1675531Z 
2022-12-01T10:43:19.1675603Z Running tests...
2022-12-01T10:43:19.1675893Z ----------------------------------------------------------------------
2022-12-01T10:43:19.1676299Z   test_Backend_enum_class (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28761
2022-12-01T10:43:19.1676698Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28762
2022-12-01T10:43:19.1677030Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 28763
2022-12-01T10:43:19.1677472Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.1677803Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.1678229Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.1678577Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.1678903Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:43:19.1679376Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.1679702Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.1680117Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.1680464Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.1680802Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:43:19.1681297Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.1681780Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.1682110Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.1682540Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.1682889Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.1683209Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:43:19.1683699Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.1684218Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.1684595Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:43:19.1684946Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:43:19.1685323Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:43:19.1685605Z ok (1.755s)
2022-12-01T10:43:19.1685696Z 
2022-12-01T10:43:19.1685930Z ----------------------------------------------------------------------
2022-12-01T10:43:19.1686170Z Ran 1 test in 1.755s
2022-12-01T10:43:19.1686285Z 
2022-12-01T10:43:19.1686347Z OK
2022-12-01T10:43:19.1686440Z 
2022-12-01T10:43:19.1686511Z Generating XML reports...
2022-12-01T10:43:19.1686977Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103108.xml
2022-12-01T10:43:19.1687470Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.1687934Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.1688254Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.1688684Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.1689030Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.1689193Z 
2022-12-01T10:43:19.1689252Z Running tests...
2022-12-01T10:43:19.1689558Z ----------------------------------------------------------------------
2022-12-01T10:43:19.1690439Z   test_DistributedDataParallel (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77317 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.240s)
2022-12-01T10:43:19.1690996Z 
2022-12-01T10:43:19.1691195Z ----------------------------------------------------------------------
2022-12-01T10:43:19.1691438Z Ran 1 test in 0.240s
2022-12-01T10:43:19.1691551Z 
2022-12-01T10:43:19.1691609Z OK (skipped=1)
2022-12-01T10:43:19.1691720Z 
2022-12-01T10:43:19.1691806Z Generating XML reports...
2022-12-01T10:43:19.1692269Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103112.xml
2022-12-01T10:43:19.1692767Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.1693214Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.1693625Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.1694062Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.1694397Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.1694567Z 
2022-12-01T10:43:19.1694640Z Running tests...
2022-12-01T10:43:19.1694941Z ----------------------------------------------------------------------
2022-12-01T10:43:19.1695371Z   test_DistributedDataParallelCPU (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28824
2022-12-01T10:43:19.1695784Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28825
2022-12-01T10:43:19.1696119Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 28826
2022-12-01T10:43:19.1696574Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.1696908Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.1697321Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.1697664Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.1698002Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:43:19.1698502Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.1698866Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.1699296Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.1699641Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.1699964Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:43:19.1700584Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.1700922Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.1701338Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.1701687Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.1702031Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:43:19.1702530Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.1703036Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.1703550Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.1703942Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:43:19.1704294Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:43:19.1704650Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_iy824zw
2022-12-01T10:43:19.1705040Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppwn32ajf
2022-12-01T10:43:19.1705437Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_iy824zw/_remote_module_non_scriptable.py
2022-12-01T10:43:19.1705839Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppwn32ajf/_remote_module_non_scriptable.py
2022-12-01T10:43:19.1706201Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:43:19.1706571Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpih0ku3mn
2022-12-01T10:43:19.1706968Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpih0ku3mn/_remote_module_non_scriptable.py
2022-12-01T10:43:19.1707334Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:43:19.1707691Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:43:19.1708048Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:43:19.1708310Z ok (1.954s)
2022-12-01T10:43:19.1708402Z 
2022-12-01T10:43:19.1708602Z ----------------------------------------------------------------------
2022-12-01T10:43:19.1708843Z Ran 1 test in 1.954s
2022-12-01T10:43:19.1708957Z 
2022-12-01T10:43:19.1709018Z OK
2022-12-01T10:43:19.1709109Z 
2022-12-01T10:43:19.1709180Z Generating XML reports...
2022-12-01T10:43:19.1709638Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103113.xml
2022-12-01T10:43:19.1710133Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.1710596Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.1710917Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.1711413Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.1711802Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.1711967Z 
2022-12-01T10:43:19.1712027Z Running tests...
2022-12-01T10:43:19.1712335Z ----------------------------------------------------------------------
2022-12-01T10:43:19.1712778Z   test_DistributedDataParallelCPU_grad_is_view (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28877
2022-12-01T10:43:19.1713224Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28878
2022-12-01T10:43:19.1713544Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 28879
2022-12-01T10:43:19.1713995Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.1714325Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.1714745Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.1715094Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.1715431Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:43:19.1715898Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.1716211Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.1716640Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.1716985Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.1717324Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:43:19.1717779Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.1718110Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.1718539Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.1718870Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.1719208Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:43:19.1719705Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.1720232Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.1720741Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.1721130Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:43:19.1721479Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:43:19.1721853Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjp6cj_az
2022-12-01T10:43:19.1722230Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy177h24v
2022-12-01T10:43:19.1722630Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjp6cj_az/_remote_module_non_scriptable.py
2022-12-01T10:43:19.1723030Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy177h24v/_remote_module_non_scriptable.py
2022-12-01T10:43:19.1723390Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:43:19.1723754Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3snj01iy
2022-12-01T10:43:19.1724188Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3snj01iy/_remote_module_non_scriptable.py
2022-12-01T10:43:19.1724596Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:43:19.1724946Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:43:19.1725302Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:43:19.1725559Z ok (1.953s)
2022-12-01T10:43:19.1725663Z 
2022-12-01T10:43:19.1725863Z ----------------------------------------------------------------------
2022-12-01T10:43:19.1726092Z Ran 1 test in 1.954s
2022-12-01T10:43:19.1726205Z 
2022-12-01T10:43:19.1726265Z OK
2022-12-01T10:43:19.1726357Z 
2022-12-01T10:43:19.1726442Z Generating XML reports...
2022-12-01T10:43:19.1726887Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103117.xml
2022-12-01T10:43:19.1727388Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.1727856Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.1728177Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.1728607Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.1728952Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.1729118Z 
2022-12-01T10:43:19.1729191Z Running tests...
2022-12-01T10:43:19.1729478Z ----------------------------------------------------------------------
2022-12-01T10:43:19.1729909Z   test_DistributedDataParallel_SyncBatchNorm (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28930
2022-12-01T10:43:19.1730345Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28931
2022-12-01T10:43:19.1730684Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 28932
2022-12-01T10:43:19.1731120Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.1731449Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.1731877Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.1732207Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.1732541Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:43:19.1733009Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.1733415Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.1733844Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.1734197Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.1734537Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:43:19.1735006Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.1735321Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.1735805Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.1736156Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.1736478Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:43:19.1737041Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.1737598Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.1738116Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.1738491Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:43:19.1738844Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:43:19.1739192Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:43:19.1739472Z skip: CUDA is not available. (1.853s)
2022-12-01T10:43:19.1739592Z 
2022-12-01T10:43:19.1739792Z ----------------------------------------------------------------------
2022-12-01T10:43:19.1740035Z Ran 1 test in 1.854s
2022-12-01T10:43:19.1740150Z 
2022-12-01T10:43:19.1740227Z OK (skipped=1)
2022-12-01T10:43:19.1740335Z 
2022-12-01T10:43:19.1740519Z Generating XML reports...
2022-12-01T10:43:19.1740983Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103120.xml
2022-12-01T10:43:19.1741480Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.1741945Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.1742265Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.1742694Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.1743041Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.1743206Z 
2022-12-01T10:43:19.1743265Z Running tests...
2022-12-01T10:43:19.1743570Z ----------------------------------------------------------------------
2022-12-01T10:43:19.1744012Z   test_DistributedDataParallel_SyncBatchNorm_2D_Input (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28983
2022-12-01T10:43:19.1744449Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28984
2022-12-01T10:43:19.1744766Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 28985
2022-12-01T10:43:19.1745214Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.1745544Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.1745962Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.1746306Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.1746644Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:43:19.1747117Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.1747438Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.1747870Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.1748213Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.1748548Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:43:19.1749003Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.1749333Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.1749760Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.1750150Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.1750525Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:43:19.1751024Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.1751548Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.1752053Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.1752446Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:43:19.1752797Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:43:19.1753150Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:43:19.1753418Z skip: CUDA is not available. (1.858s)
2022-12-01T10:43:19.1753554Z 
2022-12-01T10:43:19.1753754Z ----------------------------------------------------------------------
2022-12-01T10:43:19.1754000Z Ran 1 test in 1.858s
2022-12-01T10:43:19.1754114Z 
2022-12-01T10:43:19.1754175Z OK (skipped=1)
2022-12-01T10:43:19.1754283Z 
2022-12-01T10:43:19.1754366Z Generating XML reports...
2022-12-01T10:43:19.1754825Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103123.xml
2022-12-01T10:43:19.1755319Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.1755768Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.1756099Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.1756528Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.1756862Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.1757026Z 
2022-12-01T10:43:19.1757097Z Running tests...
2022-12-01T10:43:19.1757400Z ----------------------------------------------------------------------
2022-12-01T10:43:19.1757852Z   test_DistributedDataParallel_SyncBatchNorm_Channels_Last (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29036
2022-12-01T10:43:19.1758282Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29037
2022-12-01T10:43:19.1758613Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29038
2022-12-01T10:43:19.1759063Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.1759394Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.1759809Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.1760156Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.1760495Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:43:19.1760951Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.1761278Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.1761702Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.1762048Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.1762370Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:43:19.1762881Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.1763246Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.1763661Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.1764003Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.1764338Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:43:19.1764836Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.1765339Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.1765852Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.1766242Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:43:19.1766593Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:43:19.1766926Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:43:19.1767202Z skip: CUDA is not available. (1.752s)
2022-12-01T10:43:19.1767334Z 
2022-12-01T10:43:19.1767529Z ----------------------------------------------------------------------
2022-12-01T10:43:19.1767759Z Ran 1 test in 1.752s
2022-12-01T10:43:19.1767871Z 
2022-12-01T10:43:19.1767944Z OK (skipped=1)
2022-12-01T10:43:19.1768051Z 
2022-12-01T10:43:19.1768135Z Generating XML reports...
2022-12-01T10:43:19.1768594Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103127.xml
2022-12-01T10:43:19.1769073Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.1769532Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.1769867Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.1770280Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.1770625Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.1770788Z 
2022-12-01T10:43:19.1770860Z Running tests...
2022-12-01T10:43:19.1771160Z ----------------------------------------------------------------------
2022-12-01T10:43:19.1771608Z   test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29089
2022-12-01T10:43:19.1772066Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29090
2022-12-01T10:43:19.1772399Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29091
2022-12-01T10:43:19.1772848Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.1773162Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.1773669Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.1774018Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.1774343Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:43:19.1774816Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.1775149Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.1775626Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.1775988Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.1776322Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:43:19.1776790Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.1777119Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.1777531Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.1777879Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.1778219Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:43:19.1778705Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.1779230Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.1779747Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.1780134Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:43:19.1780609Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:43:19.1780955Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:43:19.1781239Z skip: CUDA is not available. (1.853s)
2022-12-01T10:43:19.1781372Z 
2022-12-01T10:43:19.1781571Z ----------------------------------------------------------------------
2022-12-01T10:43:19.1781802Z Ran 1 test in 1.853s
2022-12-01T10:43:19.1781918Z 
2022-12-01T10:43:19.1781994Z OK (skipped=1)
2022-12-01T10:43:19.1782105Z 
2022-12-01T10:43:19.1782192Z Generating XML reports...
2022-12-01T10:43:19.1782645Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103130.xml
2022-12-01T10:43:19.1783142Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.1783608Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.1783948Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.1784365Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.1784712Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.1784875Z 
2022-12-01T10:43:19.1784948Z Running tests...
2022-12-01T10:43:19.1785239Z ----------------------------------------------------------------------
2022-12-01T10:43:19.1785700Z   test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29142
2022-12-01T10:43:19.1786158Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29143
2022-12-01T10:43:19.1786488Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29144
2022-12-01T10:43:19.1786927Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.1787255Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.1787684Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.1788028Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.1788416Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:43:19.1788937Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.1789271Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.1789686Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.1790034Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.1790371Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:43:19.1790838Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.1791153Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.1791585Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.1791929Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.1792250Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:43:19.1792745Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.1793263Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.1793775Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.1794141Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:43:19.1794483Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:43:19.1794839Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:43:19.1795119Z skip: CUDA is not available. (1.755s)
2022-12-01T10:43:19.1795238Z 
2022-12-01T10:43:19.1795433Z ----------------------------------------------------------------------
2022-12-01T10:43:19.1795671Z Ran 1 test in 1.755s
2022-12-01T10:43:19.1795783Z 
2022-12-01T10:43:19.1795854Z OK (skipped=1)
2022-12-01T10:43:19.1795963Z 
2022-12-01T10:43:19.1796033Z Generating XML reports...
2022-12-01T10:43:19.1796492Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103133.xml
2022-12-01T10:43:19.1796988Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.1797451Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.1797767Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.1798199Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.1798547Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.1798710Z 
2022-12-01T10:43:19.1798769Z Running tests...
2022-12-01T10:43:19.1799068Z ----------------------------------------------------------------------
2022-12-01T10:43:19.1799507Z   test_DistributedDataParallel_SyncBatchNorm_No_Affine (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29195
2022-12-01T10:43:19.1799942Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29196
2022-12-01T10:43:19.1800262Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29197
2022-12-01T10:43:19.1800704Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.1801069Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.1801529Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.1801890Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.1802227Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:43:19.1802701Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.1803016Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.1803446Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.1803795Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.1804133Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:43:19.1804586Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.1804917Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.1805344Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.1805688Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.1806011Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:43:19.1806508Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.1807027Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.1807538Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.1807928Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:43:19.1808277Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:43:19.1808625Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:43:19.1808892Z skip: CUDA is not available. (1.853s)
2022-12-01T10:43:19.1809025Z 
2022-12-01T10:43:19.1809223Z ----------------------------------------------------------------------
2022-12-01T10:43:19.1809465Z Ran 1 test in 1.853s
2022-12-01T10:43:19.1809581Z 
2022-12-01T10:43:19.1809640Z OK (skipped=1)
2022-12-01T10:43:19.1809749Z 
2022-12-01T10:43:19.1809834Z Generating XML reports...
2022-12-01T10:43:19.1810301Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103136.xml
2022-12-01T10:43:19.1810799Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.1811251Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.1811582Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.1812012Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.1812359Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.1812511Z 
2022-12-01T10:43:19.1812583Z Running tests...
2022-12-01T10:43:19.1812884Z ----------------------------------------------------------------------
2022-12-01T10:43:19.1813436Z   test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29248
2022-12-01T10:43:19.1813938Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29249
2022-12-01T10:43:19.1814278Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29250
2022-12-01T10:43:19.1814775Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.1815110Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.1815528Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.1815875Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.1816217Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:43:19.1816677Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.1817004Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.1817435Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.1817782Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.1818103Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:43:19.1818569Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.1818899Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.1819327Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.1819656Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.1819990Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:43:19.1820592Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.1821108Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.1821626Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.1822013Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:43:19.1822364Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:43:19.1822697Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:43:19.1822973Z skip: CUDA is not available. (1.754s)
2022-12-01T10:43:19.1823106Z 
2022-12-01T10:43:19.1823302Z ----------------------------------------------------------------------
2022-12-01T10:43:19.1823545Z Ran 1 test in 1.755s
2022-12-01T10:43:19.1823645Z 
2022-12-01T10:43:19.1823718Z OK (skipped=1)
2022-12-01T10:43:19.1823827Z 
2022-12-01T10:43:19.1823911Z Generating XML reports...
2022-12-01T10:43:19.1824371Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103140.xml
2022-12-01T10:43:19.1824853Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.1825314Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.1825650Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.1826081Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.1826415Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.1826578Z 
2022-12-01T10:43:19.1826650Z Running tests...
2022-12-01T10:43:19.1827023Z ----------------------------------------------------------------------
2022-12-01T10:43:19.1827954Z   test_DistributedDataParallel_non_default_stream (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/76428 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.235s)
2022-12-01T10:43:19.1828461Z 
2022-12-01T10:43:19.1828646Z ----------------------------------------------------------------------
2022-12-01T10:43:19.1828886Z Ran 1 test in 0.235s
2022-12-01T10:43:19.1828997Z 
2022-12-01T10:43:19.1829068Z OK (skipped=1)
2022-12-01T10:43:19.1829175Z 
2022-12-01T10:43:19.1829263Z Generating XML reports...
2022-12-01T10:43:19.1829717Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103143.xml
2022-12-01T10:43:19.1830222Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.1830689Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.1831010Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.1831443Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.1831794Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.1831960Z 
2022-12-01T10:43:19.1832033Z Running tests...
2022-12-01T10:43:19.1832323Z ----------------------------------------------------------------------
2022-12-01T10:43:19.1832753Z   test_DistributedDataParallel_requires_grad (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29311
2022-12-01T10:43:19.1833189Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29312
2022-12-01T10:43:19.1833506Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29313
2022-12-01T10:43:19.1833962Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.1834293Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.1834718Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.1835052Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.1835389Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:43:19.1835858Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.1836189Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.1836607Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.1836956Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.1837290Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:43:19.1837743Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.1838068Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.1838492Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.1838835Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.1839156Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:43:19.1839697Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.1840254Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.1840769Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.1841141Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:43:19.1841492Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:43:19.1841833Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:43:19.1842072Z ok (1.952s)
2022-12-01T10:43:19.1842175Z 
2022-12-01T10:43:19.1842370Z ----------------------------------------------------------------------
2022-12-01T10:43:19.1842615Z Ran 1 test in 1.952s
2022-12-01T10:43:19.1842729Z 
2022-12-01T10:43:19.1842799Z OK
2022-12-01T10:43:19.1842880Z 
2022-12-01T10:43:19.1842966Z Generating XML reports...
2022-12-01T10:43:19.1843429Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103145.xml
2022-12-01T10:43:19.1843925Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.1844376Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.1844708Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.1845135Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.1845485Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.1845637Z 
2022-12-01T10:43:19.1845711Z Running tests...
2022-12-01T10:43:19.1846010Z ----------------------------------------------------------------------
2022-12-01T10:43:19.1846915Z   test_DistributedDataParallel_with_amp_and_grad_is_view (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77294 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.236s)
2022-12-01T10:43:19.1847396Z 
2022-12-01T10:43:19.1847589Z ----------------------------------------------------------------------
2022-12-01T10:43:19.1847818Z Ran 1 test in 0.237s
2022-12-01T10:43:19.1847931Z 
2022-12-01T10:43:19.1848005Z OK (skipped=1)
2022-12-01T10:43:19.1848113Z 
2022-12-01T10:43:19.1848196Z Generating XML reports...
2022-12-01T10:43:19.1848652Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103148.xml
2022-12-01T10:43:19.1849137Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.1849601Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.1849938Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.1850354Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.1850699Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.1850862Z 
2022-12-01T10:43:19.1850935Z Running tests...
2022-12-01T10:43:19.1851234Z ----------------------------------------------------------------------
2022-12-01T10:43:19.1851632Z   test_DistributedSampler_padding (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29374
2022-12-01T10:43:19.1852047Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29375
2022-12-01T10:43:19.1852424Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29376
2022-12-01T10:43:19.1852895Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.1853226Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.1853739Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.1854087Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.1854413Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:43:19.1854912Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.1855411Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.1855742Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.1856159Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.1856507Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.1856844Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:43:19.1857296Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.1857624Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.1858054Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.1858397Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.1858720Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:43:19.1859213Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.1859735Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.1860123Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:43:19.1860636Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:43:19.1860981Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:43:19.1861268Z skip: Need at least 3 CUDA devices (1.858s)
2022-12-01T10:43:19.1861407Z 
2022-12-01T10:43:19.1861595Z ----------------------------------------------------------------------
2022-12-01T10:43:19.1861836Z Ran 1 test in 1.858s
2022-12-01T10:43:19.1861952Z 
2022-12-01T10:43:19.1862024Z OK (skipped=1)
2022-12-01T10:43:19.1862135Z 
2022-12-01T10:43:19.1862221Z Generating XML reports...
2022-12-01T10:43:19.1862673Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103150.xml
2022-12-01T10:43:19.1863167Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.1863629Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.1863950Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.1864378Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.1864720Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.1864885Z 
2022-12-01T10:43:19.1864957Z Running tests...
2022-12-01T10:43:19.1865245Z ----------------------------------------------------------------------
2022-12-01T10:43:19.1865671Z   test_SyncBatchNorm_process_group (__main__.TestDistBackendWithSpawn) ... skip: no torchvision (0.002s)
2022-12-01T10:43:19.1865925Z 
2022-12-01T10:43:19.1866120Z ----------------------------------------------------------------------
2022-12-01T10:43:19.1866347Z Ran 1 test in 0.002s
2022-12-01T10:43:19.1866460Z 
2022-12-01T10:43:19.1866533Z OK (skipped=1)
2022-12-01T10:43:19.1866642Z 
2022-12-01T10:43:19.1866727Z Generating XML reports...
2022-12-01T10:43:19.1867186Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103153.xml
2022-12-01T10:43:19.1867662Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.1868122Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.1868457Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.1868887Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.1869225Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.1869390Z 
2022-12-01T10:43:19.1869462Z Running tests...
2022-12-01T10:43:19.1869761Z ----------------------------------------------------------------------
2022-12-01T10:43:19.1870065Z   test_accumulate_gradients_no_sync (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1870462Z Runs _test_accumulate_gradients_no_sync using default inputs ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29437
2022-12-01T10:43:19.1870857Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29438
2022-12-01T10:43:19.1871189Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29439
2022-12-01T10:43:19.1871626Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.1871957Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.1872391Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.1872720Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.1873056Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:43:19.1873551Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.1874051Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.1874365Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.1874791Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.1875136Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.1875473Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:43:19.1875927Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.1876252Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.1876679Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.1877006Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.1877342Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:43:19.1877834Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.1878384Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.1878787Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:43:19.1879132Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:43:19.1879479Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:43:19.1879848Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx55q89oe
2022-12-01T10:43:19.1880227Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd7spa2ts
2022-12-01T10:43:19.1880614Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz8kh2j59
2022-12-01T10:43:19.1881011Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx55q89oe/_remote_module_non_scriptable.py
2022-12-01T10:43:19.1881405Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd7spa2ts/_remote_module_non_scriptable.py
2022-12-01T10:43:19.1881810Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz8kh2j59/_remote_module_non_scriptable.py
2022-12-01T10:43:19.1882091Z ok (1.752s)
2022-12-01T10:43:19.1882195Z 
2022-12-01T10:43:19.1882406Z ----------------------------------------------------------------------
2022-12-01T10:43:19.1882634Z Ran 1 test in 1.752s
2022-12-01T10:43:19.1882746Z 
2022-12-01T10:43:19.1882806Z OK
2022-12-01T10:43:19.1882897Z 
2022-12-01T10:43:19.1882980Z Generating XML reports...
2022-12-01T10:43:19.1883423Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103154.xml
2022-12-01T10:43:19.1883919Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.1884380Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.1884714Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.1885132Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.1885477Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.1885639Z 
2022-12-01T10:43:19.1885711Z Running tests...
2022-12-01T10:43:19.1885998Z ----------------------------------------------------------------------
2022-12-01T10:43:19.1886332Z   test_accumulate_gradients_no_sync_allreduce_hook (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1886747Z Runs multiple iterations on _test_accumulate_gradients_no_sync ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29490
2022-12-01T10:43:19.1887149Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29491
2022-12-01T10:43:19.1887468Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29492
2022-12-01T10:43:19.1887919Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.1888254Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.1888671Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.1889016Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.1889352Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:43:19.1889820Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.1890136Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.1890560Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.1890939Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.1891277Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:43:19.1891763Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.1892091Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.1892514Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.1892840Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.1893174Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:43:19.1893772Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.1894301Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.1894807Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.1895196Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:43:19.1895545Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:43:19.1895896Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:43:19.1896254Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpoomdnttp
2022-12-01T10:43:19.1896648Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6k3qg9wk
2022-12-01T10:43:19.1897032Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqde6bybz
2022-12-01T10:43:19.1897416Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpoomdnttp/_remote_module_non_scriptable.py
2022-12-01T10:43:19.1897822Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6k3qg9wk/_remote_module_non_scriptable.py
2022-12-01T10:43:19.1898223Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqde6bybz/_remote_module_non_scriptable.py
2022-12-01T10:43:19.1898606Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:43:19.1898951Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:43:19.1899313Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:43:19.1899570Z ok (1.752s)
2022-12-01T10:43:19.1899674Z 
2022-12-01T10:43:19.1899861Z ----------------------------------------------------------------------
2022-12-01T10:43:19.1900101Z Ran 1 test in 1.753s
2022-12-01T10:43:19.1900213Z 
2022-12-01T10:43:19.1900274Z OK
2022-12-01T10:43:19.1900476Z 
2022-12-01T10:43:19.1900565Z Generating XML reports...
2022-12-01T10:43:19.1901022Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103158.xml
2022-12-01T10:43:19.1901522Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.1901988Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.1902307Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.1902743Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.1903093Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.1903257Z 
2022-12-01T10:43:19.1903328Z Running tests...
2022-12-01T10:43:19.1903617Z ----------------------------------------------------------------------
2022-12-01T10:43:19.1904025Z   test_accumulate_gradients_no_sync_allreduce_with_then_hook (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1904512Z Runs multiple iterations on _test_accumulate_gradients_no_sync using allreduce ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29543
2022-12-01T10:43:19.1904923Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29544
2022-12-01T10:43:19.1905244Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29545
2022-12-01T10:43:19.1905694Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.1906030Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.1906445Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.1906794Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.1907137Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:43:19.1907609Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.1907923Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.1908349Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.1908691Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.1909015Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:43:19.1909479Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.1909802Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.1910227Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.1910555Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.1910891Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:43:19.1911387Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.1911907Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.1912417Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.1912806Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:43:19.1913196Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:43:19.1913542Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:43:19.1913906Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprqd1spdk
2022-12-01T10:43:19.1914295Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprlbw9ijq
2022-12-01T10:43:19.1914677Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa8s93yd2
2022-12-01T10:43:19.1915059Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprqd1spdk/_remote_module_non_scriptable.py
2022-12-01T10:43:19.1915463Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprlbw9ijq/_remote_module_non_scriptable.py
2022-12-01T10:43:19.1915865Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa8s93yd2/_remote_module_non_scriptable.py
2022-12-01T10:43:19.1916241Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:43:19.1916641Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:43:19.1917003Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:43:19.1917290Z ok (1.755s)
2022-12-01T10:43:19.1917394Z 
2022-12-01T10:43:19.1917582Z ----------------------------------------------------------------------
2022-12-01T10:43:19.1917822Z Ran 1 test in 1.756s
2022-12-01T10:43:19.1917934Z 
2022-12-01T10:43:19.1917995Z OK
2022-12-01T10:43:19.1918088Z 
2022-12-01T10:43:19.1918173Z Generating XML reports...
2022-12-01T10:43:19.1918621Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103201.xml
2022-12-01T10:43:19.1919117Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.1919581Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.1919901Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.1920332Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.1920676Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.1920841Z 
2022-12-01T10:43:19.1920912Z Running tests...
2022-12-01T10:43:19.1921220Z ----------------------------------------------------------------------
2022-12-01T10:43:19.1921550Z   test_accumulate_gradients_no_sync_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-12-01T10:43:19.1921964Z Runs _test_accumulate_gradients_no_sync using default inputs ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29596
2022-12-01T10:43:19.1922341Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29597
2022-12-01T10:43:19.1922674Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29598
2022-12-01T10:43:19.1923126Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.1923458Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.1923875Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.1924218Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.1924554Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:43:19.1925028Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.1925342Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.1925768Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.1926112Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.1926439Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:43:19.1926910Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.1927240Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.1927667Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.1927994Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.1928329Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:43:19.1928826Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.1929348Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.1929886Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.1930304Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:43:19.1930652Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:43:19.1931012Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbnzex8_m
2022-12-01T10:43:19.1931418Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbnzex8_m/_remote_module_non_scriptable.py
2022-12-01T10:43:19.1931823Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0vm6eghp
2022-12-01T10:43:19.1932226Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0vm6eghp/_remote_module_non_scriptable.py
2022-12-01T10:43:19.1932590Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:43:19.1932964Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9arlulko
2022-12-01T10:43:19.1933449Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9arlulko/_remote_module_non_scriptable.py
2022-12-01T10:43:19.1933722Z ok (1.953s)
2022-12-01T10:43:19.1933828Z 
2022-12-01T10:43:19.1934036Z ----------------------------------------------------------------------
2022-12-01T10:43:19.1934112Z Ran 1 test in 1.954s
2022-12-01T10:43:19.1934117Z 
2022-12-01T10:43:19.1934179Z OK
2022-12-01T10:43:19.1934184Z 
2022-12-01T10:43:19.1934268Z Generating XML reports...
2022-12-01T10:43:19.1934614Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103204.xml
2022-12-01T10:43:19.1934847Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.1935126Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.1935239Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.1935523Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.1935663Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.1935668Z 
2022-12-01T10:43:19.1935743Z Running tests...
2022-12-01T10:43:19.1935940Z ----------------------------------------------------------------------
2022-12-01T10:43:19.1936167Z   test_all_gather (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29649
2022-12-01T10:43:19.1936324Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29650
2022-12-01T10:43:19.1936479Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29651
2022-12-01T10:43:19.1936740Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.1936867Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.1937146Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.1937281Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.1937459Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:43:19.1937730Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.1937854Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.1938133Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.1938267Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.1938471Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:43:19.1938769Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.1938892Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.1939171Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.1939304Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.1939481Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:43:19.1939782Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.1940077Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.1940509Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.1940665Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:43:19.1940917Z STAGE:2022-12-01 10:32:09 29650:29650 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.1941080Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:43:19.1941244Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:43:19.1941493Z STAGE:2022-12-01 10:32:09 29649:29649 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.1941736Z STAGE:2022-12-01 10:32:09 29651:29651 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.1941986Z STAGE:2022-12-01 10:32:09 29649:29649 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.1942238Z STAGE:2022-12-01 10:32:09 29651:29651 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.1942467Z STAGE:2022-12-01 10:32:09 29650:29650 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.1942712Z STAGE:2022-12-01 10:32:09 29649:29649 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.1942950Z STAGE:2022-12-01 10:32:09 29651:29651 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.1943192Z STAGE:2022-12-01 10:32:09 29650:29650 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.1943440Z STAGE:2022-12-01 10:32:09 29650:29650 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.1943683Z STAGE:2022-12-01 10:32:09 29651:29651 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.1943925Z STAGE:2022-12-01 10:32:09 29649:29649 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.1944164Z STAGE:2022-12-01 10:32:09 29651:29651 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.1944408Z STAGE:2022-12-01 10:32:09 29650:29650 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.1944634Z STAGE:2022-12-01 10:32:09 29649:29649 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.1945043Z STAGE:2022-12-01 10:32:09 29650:29650 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 10:32:09 29649:29649 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.1945288Z STAGE:2022-12-01 10:32:09 29651:29651 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.1945293Z 
2022-12-01T10:43:19.1945362Z ok (1.955s)
2022-12-01T10:43:19.1945367Z 
2022-12-01T10:43:19.1945563Z ----------------------------------------------------------------------
2022-12-01T10:43:19.1945637Z Ran 1 test in 1.955s
2022-12-01T10:43:19.1945692Z 
2022-12-01T10:43:19.1945755Z OK
2022-12-01T10:43:19.1945800Z 
2022-12-01T10:43:19.1945889Z Generating XML reports...
2022-12-01T10:43:19.1946233Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103207.xml
2022-12-01T10:43:19.1946465Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.1946727Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.1946852Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.1947134Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.1947272Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.1947278Z 
2022-12-01T10:43:19.1947351Z Running tests...
2022-12-01T10:43:19.1947552Z ----------------------------------------------------------------------
2022-12-01T10:43:19.1947799Z   test_all_gather_coalesced_complex (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29705
2022-12-01T10:43:19.1947959Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29706
2022-12-01T10:43:19.1948100Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29707
2022-12-01T10:43:19.1948377Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.1948500Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.1948780Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.1948914Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.1949091Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:43:19.1949364Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.1949487Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.1949766Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.1949887Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.1950063Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:43:19.1950363Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.1950634Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.1950760Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.1951046Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.1951185Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.1951360Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:43:19.1951654Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.1951938Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.1952101Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:43:19.1952265Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:43:19.1952462Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:43:19.1952743Z STAGE:2022-12-01 10:32:12 29705:29705 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.1952984Z STAGE:2022-12-01 10:32:12 29707:29707 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.1953226Z STAGE:2022-12-01 10:32:12 29706:29706 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.1953635Z STAGE:2022-12-01 10:32:12 29706:29706 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 10:32:12 29705:29705 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.1953641Z 
2022-12-01T10:43:19.1953891Z STAGE:2022-12-01 10:32:12 29707:29707 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.1954288Z STAGE:2022-12-01 10:32:12 29705:29705 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-12-01 10:32:12 29706:29706 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.1954293Z 
2022-12-01T10:43:19.1954524Z STAGE:2022-12-01 10:32:12 29707:29707 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.1954935Z STAGE:2022-12-01 10:32:12 29705:29705 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 10:32:12 29706:29706 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.1954940Z 
2022-12-01T10:43:19.1955187Z STAGE:2022-12-01 10:32:12 29707:29707 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.1955425Z STAGE:2022-12-01 10:32:12 29705:29705 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.1955666Z STAGE:2022-12-01 10:32:12 29706:29706 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.1955906Z STAGE:2022-12-01 10:32:12 29707:29707 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.1956153Z STAGE:2022-12-01 10:32:12 29705:29705 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.1956400Z STAGE:2022-12-01 10:32:12 29706:29706 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.1956645Z STAGE:2022-12-01 10:32:12 29707:29707 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.1957218Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:2442: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-12-01T10:43:19.1957373Z   "torch.distributed.all_gather_coalesced will be deprecated. If you must "
2022-12-01T10:43:19.1957922Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:2442: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-12-01T10:43:19.1958078Z   "torch.distributed.all_gather_coalesced will be deprecated. If you must "
2022-12-01T10:43:19.1958635Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:2442: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-12-01T10:43:19.1958784Z   "torch.distributed.all_gather_coalesced will be deprecated. If you must "
2022-12-01T10:43:19.1958850Z ok (1.852s)
2022-12-01T10:43:19.1958856Z 
2022-12-01T10:43:19.1959053Z ----------------------------------------------------------------------
2022-12-01T10:43:19.1959129Z Ran 1 test in 1.852s
2022-12-01T10:43:19.1959134Z 
2022-12-01T10:43:19.1959193Z OK
2022-12-01T10:43:19.1959198Z 
2022-12-01T10:43:19.1959282Z Generating XML reports...
2022-12-01T10:43:19.1959648Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103211.xml
2022-12-01T10:43:19.1959909Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.1960181Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.1960308Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.1960587Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.1960723Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.1960729Z 
2022-12-01T10:43:19.1960800Z Running tests...
2022-12-01T10:43:19.1960995Z ----------------------------------------------------------------------
2022-12-01T10:43:19.1961247Z   test_all_gather_coalesced_full_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29761
2022-12-01T10:43:19.1961389Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29762
2022-12-01T10:43:19.1961543Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29763
2022-12-01T10:43:19.1961816Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.1961939Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.1962218Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.1962353Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.1962532Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:43:19.1962800Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.1962924Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.1963189Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.1963323Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.1963499Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:43:19.1963765Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.1963887Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.1964166Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.1964298Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.1964474Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:43:19.1964760Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.1965058Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.1965353Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.1965516Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:43:19.1965680Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:43:19.1965835Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:43:19.1966011Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T10:43:19.1966208Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T10:43:19.1966408Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-12-01T10:43:19.1966690Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:43:19.1966983Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:43:19.1967277Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:43:19.1967526Z STAGE:2022-12-01 10:32:16 29762:29762 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.1967770Z STAGE:2022-12-01 10:32:16 29761:29761 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.1968016Z STAGE:2022-12-01 10:32:16 29763:29763 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.1968263Z STAGE:2022-12-01 10:32:16 29762:29762 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.1968514Z STAGE:2022-12-01 10:32:16 29763:29763 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.1968758Z STAGE:2022-12-01 10:32:16 29761:29761 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.1968985Z STAGE:2022-12-01 10:32:16 29762:29762 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.1969232Z STAGE:2022-12-01 10:32:16 29761:29761 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.1969474Z STAGE:2022-12-01 10:32:16 29763:29763 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.1969718Z STAGE:2022-12-01 10:32:16 29762:29762 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.1969967Z STAGE:2022-12-01 10:32:16 29761:29761 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.1970212Z STAGE:2022-12-01 10:32:16 29763:29763 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.1970454Z STAGE:2022-12-01 10:32:16 29762:29762 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.1970697Z STAGE:2022-12-01 10:32:16 29761:29761 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.1970938Z STAGE:2022-12-01 10:32:16 29763:29763 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.1971168Z STAGE:2022-12-01 10:32:16 29762:29762 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.1971579Z STAGE:2022-12-01 10:32:16 29761:29761 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 10:32:16 29763:29763 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.1971585Z 
2022-12-01T10:43:19.1972153Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:2442: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-12-01T10:43:19.1972306Z   "torch.distributed.all_gather_coalesced will be deprecated. If you must "
2022-12-01T10:43:19.1972865Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:2442: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-12-01T10:43:19.1973015Z   "torch.distributed.all_gather_coalesced will be deprecated. If you must "
2022-12-01T10:43:19.1973698Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:2442: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-12-01T10:43:19.1973879Z   "torch.distributed.all_gather_coalesced will be deprecated. If you must "
2022-12-01T10:43:19.1973948Z ok (1.851s)
2022-12-01T10:43:19.1973953Z 
2022-12-01T10:43:19.1974157Z ----------------------------------------------------------------------
2022-12-01T10:43:19.1974233Z Ran 1 test in 1.851s
2022-12-01T10:43:19.1974238Z 
2022-12-01T10:43:19.1974286Z OK
2022-12-01T10:43:19.1974304Z 
2022-12-01T10:43:19.1974374Z Generating XML reports...
2022-12-01T10:43:19.1974716Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103214.xml
2022-12-01T10:43:19.1974951Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.1975227Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.1975355Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.1975639Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.1975783Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.1975789Z 
2022-12-01T10:43:19.1975863Z Running tests...
2022-12-01T10:43:19.1976043Z ----------------------------------------------------------------------
2022-12-01T10:43:19.1976291Z   test_all_gather_coalesced_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29826
2022-12-01T10:43:19.1976450Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29827
2022-12-01T10:43:19.1976601Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29828
2022-12-01T10:43:19.1976873Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.1977001Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.1977286Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.1977425Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.1977603Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:43:19.1977862Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.1977988Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.1978265Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.1978398Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.1978574Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:43:19.1978846Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.1978969Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.1979245Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.1979365Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.1979536Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:43:19.1979832Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.1980128Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.1980571Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.1980776Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:43:19.1980942Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:43:19.1981103Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:43:19.1981280Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T10:43:19.1981436Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-12-01T10:43:19.1981604Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T10:43:19.1981903Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:43:19.1982158Z STAGE:2022-12-01 10:32:19 29827:29827 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.1982459Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:43:19.1982753Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:43:19.1982999Z STAGE:2022-12-01 10:32:19 29828:29828 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.1983419Z STAGE:2022-12-01 10:32:19 29828:29828 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 10:32:19 29827:29827 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.1983426Z 
2022-12-01T10:43:19.1983669Z STAGE:2022-12-01 10:32:19 29828:29828 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.1983912Z STAGE:2022-12-01 10:32:19 29827:29827 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.1984147Z STAGE:2022-12-01 10:32:19 29827:29827 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.1984397Z STAGE:2022-12-01 10:32:19 29828:29828 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.1984638Z STAGE:2022-12-01 10:32:19 29827:29827 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.1984875Z STAGE:2022-12-01 10:32:19 29828:29828 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.1985122Z STAGE:2022-12-01 10:32:19 29827:29827 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.1985367Z STAGE:2022-12-01 10:32:19 29828:29828 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.1985934Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:2442: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-12-01T10:43:19.1986088Z   "torch.distributed.all_gather_coalesced will be deprecated. If you must "
2022-12-01T10:43:19.1986646Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:2442: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-12-01T10:43:19.1986795Z   "torch.distributed.all_gather_coalesced will be deprecated. If you must "
2022-12-01T10:43:19.1986849Z ok (1.955s)
2022-12-01T10:43:19.1986867Z 
2022-12-01T10:43:19.1987050Z ----------------------------------------------------------------------
2022-12-01T10:43:19.1987124Z Ran 1 test in 1.955s
2022-12-01T10:43:19.1987129Z 
2022-12-01T10:43:19.1987190Z OK
2022-12-01T10:43:19.1987194Z 
2022-12-01T10:43:19.1987278Z Generating XML reports...
2022-12-01T10:43:19.1987649Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103217.xml
2022-12-01T10:43:19.1987909Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.1988179Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.1988305Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.1988573Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.1988705Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.1988711Z 
2022-12-01T10:43:19.1988783Z Running tests...
2022-12-01T10:43:19.1988980Z ----------------------------------------------------------------------
2022-12-01T10:43:19.1989228Z   test_all_gather_coalesced_simple (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29887
2022-12-01T10:43:19.1989386Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29888
2022-12-01T10:43:19.1989537Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29889
2022-12-01T10:43:19.1989810Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.1989922Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.1990200Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.1990336Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.1990511Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:43:19.1990781Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.1990903Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.1991186Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.1991319Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.1991496Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:43:19.1991754Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.1991876Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.1992152Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.1992284Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.1992459Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:43:19.1992761Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.1993056Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.1993349Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.1993511Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:43:19.1993660Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:43:19.1993817Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:43:19.1994099Z STAGE:2022-12-01 10:32:22 29887:29887 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.1994350Z STAGE:2022-12-01 10:32:22 29889:29889 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.1994622Z STAGE:2022-12-01 10:32:22 29888:29888 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.1994873Z STAGE:2022-12-01 10:32:22 29887:29887 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.1995120Z STAGE:2022-12-01 10:32:22 29889:29889 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.1995364Z STAGE:2022-12-01 10:32:22 29888:29888 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.1995601Z STAGE:2022-12-01 10:32:22 29887:29887 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.1995827Z STAGE:2022-12-01 10:32:22 29889:29889 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.1996067Z STAGE:2022-12-01 10:32:22 29888:29888 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.1996309Z STAGE:2022-12-01 10:32:22 29889:29889 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.1996556Z STAGE:2022-12-01 10:32:22 29888:29888 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.1996799Z STAGE:2022-12-01 10:32:22 29887:29887 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.1997036Z STAGE:2022-12-01 10:32:22 29888:29888 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.1997276Z STAGE:2022-12-01 10:32:22 29889:29889 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.1997511Z STAGE:2022-12-01 10:32:22 29887:29887 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.1997755Z STAGE:2022-12-01 10:32:22 29887:29887 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.1997987Z STAGE:2022-12-01 10:32:22 29889:29889 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.1998229Z STAGE:2022-12-01 10:32:22 29888:29888 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.1998796Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:2442: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-12-01T10:43:19.1998947Z   "torch.distributed.all_gather_coalesced will be deprecated. If you must "
2022-12-01T10:43:19.1999503Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:2442: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-12-01T10:43:19.1999658Z   "torch.distributed.all_gather_coalesced will be deprecated. If you must "
2022-12-01T10:43:19.2000214Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:2442: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-12-01T10:43:19.2000366Z   "torch.distributed.all_gather_coalesced will be deprecated. If you must "
2022-12-01T10:43:19.2000433Z ok (1.753s)
2022-12-01T10:43:19.2000438Z 
2022-12-01T10:43:19.2000635Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2000697Z Ran 1 test in 1.754s
2022-12-01T10:43:19.2000715Z 
2022-12-01T10:43:19.2000762Z OK
2022-12-01T10:43:19.2000766Z 
2022-12-01T10:43:19.2000849Z Generating XML reports...
2022-12-01T10:43:19.2001186Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103221.xml
2022-12-01T10:43:19.2001461Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.2001735Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2001889Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2002169Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2002304Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2002310Z 
2022-12-01T10:43:19.2002369Z Running tests...
2022-12-01T10:43:19.2002566Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2002817Z   test_all_gather_coalesced_with_empty (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29943
2022-12-01T10:43:19.2002975Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29944
2022-12-01T10:43:19.2003127Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29945
2022-12-01T10:43:19.2003407Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2003532Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2003813Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2003949Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2004115Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:43:19.2004387Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2004510Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2004792Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2004928Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2005109Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:43:19.2005374Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2005495Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2005756Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2005890Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2006066Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:43:19.2006365Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2006662Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2006958Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2007122Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:43:19.2007370Z STAGE:2022-12-01 10:32:25 29944:29944 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2007533Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:43:19.2007681Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:43:19.2007930Z STAGE:2022-12-01 10:32:26 29943:29943 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2008208Z STAGE:2022-12-01 10:32:26 29945:29945 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2008481Z STAGE:2022-12-01 10:32:26 29943:29943 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2008727Z STAGE:2022-12-01 10:32:26 29945:29945 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2008971Z STAGE:2022-12-01 10:32:26 29944:29944 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2009555Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:2442: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-12-01T10:43:19.2009708Z   "torch.distributed.all_gather_coalesced will be deprecated. If you must "
2022-12-01T10:43:19.2010264Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:2442: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-12-01T10:43:19.2010417Z   "torch.distributed.all_gather_coalesced will be deprecated. If you must "
2022-12-01T10:43:19.2010969Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:2442: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-12-01T10:43:19.2011104Z   "torch.distributed.all_gather_coalesced will be deprecated. If you must "
2022-12-01T10:43:19.2011172Z ok (1.954s)
2022-12-01T10:43:19.2011177Z 
2022-12-01T10:43:19.2011374Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2011449Z Ran 1 test in 1.954s
2022-12-01T10:43:19.2011454Z 
2022-12-01T10:43:19.2011515Z OK
2022-12-01T10:43:19.2011520Z 
2022-12-01T10:43:19.2011608Z Generating XML reports...
2022-12-01T10:43:19.2011951Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103224.xml
2022-12-01T10:43:19.2012186Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.2012463Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2012574Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2012855Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2012990Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2012995Z 
2022-12-01T10:43:19.2013066Z Running tests...
2022-12-01T10:43:19.2013338Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2013585Z   test_all_gather_complex (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29999
2022-12-01T10:43:19.2013746Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30000
2022-12-01T10:43:19.2013902Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 30001
2022-12-01T10:43:19.2014163Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2014286Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2014563Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2014702Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2014884Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:43:19.2015192Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2015345Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2015624Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2015756Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2015918Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:43:19.2016184Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2016303Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2016576Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2016706Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2016880Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:43:19.2017182Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2017477Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2017774Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2017928Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:43:19.2018176Z STAGE:2022-12-01 10:32:29 30000:30000 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2018339Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:43:19.2018501Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:43:19.2018748Z STAGE:2022-12-01 10:32:29 29999:29999 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2018990Z STAGE:2022-12-01 10:32:29 30001:30001 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2019402Z STAGE:2022-12-01 10:32:29 29999:29999 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 10:32:29 30001:30001 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2019413Z 
2022-12-01T10:43:19.2019657Z STAGE:2022-12-01 10:32:29 30000:30000 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2019897Z STAGE:2022-12-01 10:32:29 30001:30001 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2020122Z STAGE:2022-12-01 10:32:29 30000:30000 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2020457Z STAGE:2022-12-01 10:32:29 29999:29999 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2020710Z STAGE:2022-12-01 10:32:29 30000:30000 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2021123Z STAGE:2022-12-01 10:32:29 29999:29999 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 10:32:29 30001:30001 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2021129Z 
2022-12-01T10:43:19.2021368Z STAGE:2022-12-01 10:32:29 30000:30000 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2021606Z STAGE:2022-12-01 10:32:29 30001:30001 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2021845Z STAGE:2022-12-01 10:32:29 29999:29999 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2022092Z STAGE:2022-12-01 10:32:29 29999:29999 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2022572Z STAGE:2022-12-01 10:32:29 30000:30000 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 10:32:29 30001:30001 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2022614Z 
2022-12-01T10:43:19.2022684Z ok (1.953s)
2022-12-01T10:43:19.2022688Z 
2022-12-01T10:43:19.2022887Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2022950Z Ran 1 test in 1.953s
2022-12-01T10:43:19.2022954Z 
2022-12-01T10:43:19.2023016Z OK
2022-12-01T10:43:19.2023021Z 
2022-12-01T10:43:19.2023105Z Generating XML reports...
2022-12-01T10:43:19.2023448Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103227.xml
2022-12-01T10:43:19.2023686Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.2023961Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2024089Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2024373Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2024496Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2024514Z 
2022-12-01T10:43:19.2024573Z Running tests...
2022-12-01T10:43:19.2024768Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2024961Z   test_all_gather_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all gather (0.002s)
2022-12-01T10:43:19.2024966Z 
2022-12-01T10:43:19.2025159Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2025234Z Ran 1 test in 0.002s
2022-12-01T10:43:19.2025239Z 
2022-12-01T10:43:19.2025309Z OK (skipped=1)
2022-12-01T10:43:19.2025314Z 
2022-12-01T10:43:19.2025395Z Generating XML reports...
2022-12-01T10:43:19.2054959Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103231.xml
2022-12-01T10:43:19.2055358Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.2055643Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2055763Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2056046Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2056177Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2056184Z 
2022-12-01T10:43:19.2056251Z Running tests...
2022-12-01T10:43:19.2056440Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2056639Z   test_all_gather_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all gather (0.002s)
2022-12-01T10:43:19.2056645Z 
2022-12-01T10:43:19.2056835Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2056899Z Ran 1 test in 0.002s
2022-12-01T10:43:19.2056909Z 
2022-12-01T10:43:19.2056966Z OK (skipped=1)
2022-12-01T10:43:19.2056971Z 
2022-12-01T10:43:19.2057047Z Generating XML reports...
2022-12-01T10:43:19.2057381Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103232.xml
2022-12-01T10:43:19.2057602Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.2057870Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2057986Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2058258Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2058518Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2058561Z 
2022-12-01T10:43:19.2058622Z Running tests...
2022-12-01T10:43:19.2058808Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2059037Z   test_all_gather_full_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30075
2022-12-01T10:43:19.2059188Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30076
2022-12-01T10:43:19.2059335Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 30077
2022-12-01T10:43:19.2059598Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2059715Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2059987Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2060115Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2060285Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:43:19.2060820Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2060936Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2061209Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2061336Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2061506Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:43:19.2061768Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2061886Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2062156Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2062281Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2062449Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:43:19.2062739Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2063028Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2063308Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2063466Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:43:19.2063622Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:43:19.2063779Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:43:19.2063943Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T10:43:19.2064108Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-12-01T10:43:19.2064270Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T10:43:19.2064555Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:43:19.2064840Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:43:19.2065177Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:43:19.2065619Z STAGE:2022-12-01 10:32:35 30076:30076 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-12-01 10:32:35 30075:30075 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2065625Z 
2022-12-01T10:43:19.2065863Z STAGE:2022-12-01 10:32:35 30077:30077 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2066102Z STAGE:2022-12-01 10:32:35 30077:30077 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2066501Z STAGE:2022-12-01 10:32:35 30075:30075 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 10:32:35 30076:30076 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2066508Z 
2022-12-01T10:43:19.2066896Z STAGE:2022-12-01 10:32:35 30077:30077 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-12-01 10:32:35 30076:30076 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2066903Z 
2022-12-01T10:43:19.2067136Z STAGE:2022-12-01 10:32:35 30075:30075 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2067373Z STAGE:2022-12-01 10:32:35 30077:30077 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2067607Z STAGE:2022-12-01 10:32:35 30076:30076 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2067842Z STAGE:2022-12-01 10:32:35 30075:30075 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2068075Z STAGE:2022-12-01 10:32:35 30077:30077 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2068302Z STAGE:2022-12-01 10:32:35 30076:30076 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2068526Z STAGE:2022-12-01 10:32:35 30075:30075 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2068763Z STAGE:2022-12-01 10:32:35 30077:30077 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2069003Z STAGE:2022-12-01 10:32:35 30075:30075 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2069237Z STAGE:2022-12-01 10:32:35 30076:30076 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2069292Z ok (1.953s)
2022-12-01T10:43:19.2069297Z 
2022-12-01T10:43:19.2069484Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2069550Z Ran 1 test in 1.953s
2022-12-01T10:43:19.2069556Z 
2022-12-01T10:43:19.2069607Z OK
2022-12-01T10:43:19.2069612Z 
2022-12-01T10:43:19.2069686Z Generating XML reports...
2022-12-01T10:43:19.2070018Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103233.xml
2022-12-01T10:43:19.2070239Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.2070510Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2070625Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2070898Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2071026Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2071031Z 
2022-12-01T10:43:19.2071094Z Running tests...
2022-12-01T10:43:19.2071281Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2071502Z   test_all_gather_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30140
2022-12-01T10:43:19.2071651Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30141
2022-12-01T10:43:19.2071828Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 30142
2022-12-01T10:43:19.2072095Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2072233Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2072505Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2072631Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2072798Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:43:19.2073062Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2073176Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2073447Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2073575Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2073739Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:43:19.2073999Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2074113Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2074382Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2074508Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2074675Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:43:19.2074967Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2075258Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2075546Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2075697Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:43:19.2075852Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:43:19.2076016Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T10:43:19.2076167Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:43:19.2076342Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-12-01T10:43:19.2076512Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T10:43:19.2076810Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:43:19.2077112Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:43:19.2077363Z STAGE:2022-12-01 10:32:38 30141:30141 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2077645Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:43:19.2077895Z STAGE:2022-12-01 10:32:38 30142:30142 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2078141Z STAGE:2022-12-01 10:32:38 30142:30142 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2078387Z STAGE:2022-12-01 10:32:38 30141:30141 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2078659Z STAGE:2022-12-01 10:32:38 30142:30142 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2078926Z STAGE:2022-12-01 10:32:38 30141:30141 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2079338Z STAGE:2022-12-01 10:32:38 30142:30142 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 10:32:38 30141:30141 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2079345Z 
2022-12-01T10:43:19.2079414Z ok (1.953s)
2022-12-01T10:43:19.2079419Z 
2022-12-01T10:43:19.2079617Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2079694Z Ran 1 test in 1.953s
2022-12-01T10:43:19.2079699Z 
2022-12-01T10:43:19.2079745Z OK
2022-12-01T10:43:19.2079750Z 
2022-12-01T10:43:19.2079834Z Generating XML reports...
2022-12-01T10:43:19.2080178Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103237.xml
2022-12-01T10:43:19.2080418Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.2080693Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2080818Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2081099Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2081236Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2081241Z 
2022-12-01T10:43:19.2081314Z Running tests...
2022-12-01T10:43:19.2081496Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2081713Z   test_all_gather_into_cat_tensor_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_gather_into_tensor (0.002s)
2022-12-01T10:43:19.2081719Z 
2022-12-01T10:43:19.2081916Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2081992Z Ran 1 test in 0.002s
2022-12-01T10:43:19.2081999Z 
2022-12-01T10:43:19.2082072Z OK (skipped=1)
2022-12-01T10:43:19.2082077Z 
2022-12-01T10:43:19.2082161Z Generating XML reports...
2022-12-01T10:43:19.2082502Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103240.xml
2022-12-01T10:43:19.2082736Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.2082997Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2083124Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2083405Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2083539Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2083544Z 
2022-12-01T10:43:19.2083618Z Running tests...
2022-12-01T10:43:19.2083818Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2084036Z   test_all_gather_into_stack_tensor_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_gather_into_tensor (0.002s)
2022-12-01T10:43:19.2084042Z 
2022-12-01T10:43:19.2084230Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2084307Z Ran 1 test in 0.002s
2022-12-01T10:43:19.2084312Z 
2022-12-01T10:43:19.2084369Z OK (skipped=1)
2022-12-01T10:43:19.2084387Z 
2022-12-01T10:43:19.2084458Z Generating XML reports...
2022-12-01T10:43:19.2084798Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103242.xml
2022-12-01T10:43:19.2085030Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.2085338Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2085495Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2085773Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2085907Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2085912Z 
2022-12-01T10:43:19.2085985Z Running tests...
2022-12-01T10:43:19.2086168Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2086379Z   test_all_gather_multigpu (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl backend supports allgather multigpu (0.002s)
2022-12-01T10:43:19.2086384Z 
2022-12-01T10:43:19.2086573Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2086649Z Ran 1 test in 0.002s
2022-12-01T10:43:19.2086654Z 
2022-12-01T10:43:19.2086727Z OK (skipped=1)
2022-12-01T10:43:19.2086731Z 
2022-12-01T10:43:19.2086816Z Generating XML reports...
2022-12-01T10:43:19.2087160Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103243.xml
2022-12-01T10:43:19.2087393Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.2087667Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2087779Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2088061Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2088196Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2088201Z 
2022-12-01T10:43:19.2088274Z Running tests...
2022-12-01T10:43:19.2088466Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2088688Z   test_all_gather_multigpu_complex (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl backend supports allgather multigpu (0.002s)
2022-12-01T10:43:19.2088695Z 
2022-12-01T10:43:19.2088887Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2088963Z Ran 1 test in 0.002s
2022-12-01T10:43:19.2088968Z 
2022-12-01T10:43:19.2089026Z OK (skipped=1)
2022-12-01T10:43:19.2089044Z 
2022-12-01T10:43:19.2089114Z Generating XML reports...
2022-12-01T10:43:19.2089450Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103244.xml
2022-12-01T10:43:19.2089683Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.2089956Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2090081Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2090364Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2090501Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2090506Z 
2022-12-01T10:43:19.2090582Z Running tests...
2022-12-01T10:43:19.2090763Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2091006Z   test_all_gather_object_default_pg (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30241
2022-12-01T10:43:19.2091170Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30242
2022-12-01T10:43:19.2091324Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 30243
2022-12-01T10:43:19.2091595Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2091718Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2092084Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2092247Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2092426Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:43:19.2092683Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2092806Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2093086Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2093220Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2093474Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:43:19.2093754Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2093881Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2094161Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2094283Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2094459Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:43:19.2094762Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2095063Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2095360Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2095526Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:43:19.2095694Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:43:19.2095857Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:43:19.2095926Z ok (1.854s)
2022-12-01T10:43:19.2095932Z 
2022-12-01T10:43:19.2096115Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2096191Z Ran 1 test in 1.854s
2022-12-01T10:43:19.2096196Z 
2022-12-01T10:43:19.2096255Z OK
2022-12-01T10:43:19.2096260Z 
2022-12-01T10:43:19.2096345Z Generating XML reports...
2022-12-01T10:43:19.2096684Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103246.xml
2022-12-01T10:43:19.2096916Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.2097191Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2097319Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2097600Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2097722Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2097728Z 
2022-12-01T10:43:19.2097802Z Running tests...
2022-12-01T10:43:19.2097998Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2098240Z   test_all_gather_object_subgroup (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30294
2022-12-01T10:43:19.2098400Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30295
2022-12-01T10:43:19.2098592Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 30296
2022-12-01T10:43:19.2098905Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2099030Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2099296Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2099432Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2099610Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:43:19.2099878Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2100002Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2100285Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2100639Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2100821Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:43:19.2101093Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2101201Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2101484Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2101618Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2101795Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:43:19.2102093Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2102390Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2102682Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2102850Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:43:19.2103008Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:43:19.2103156Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:43:19.2103333Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T10:43:19.2103508Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T10:43:19.2103683Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-12-01T10:43:19.2103974Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:43:19.2104271Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:43:19.2104558Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:43:19.2104731Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-12-01T10:43:19.2104904Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 2
2022-12-01T10:43:19.2105062Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-12-01T10:43:19.2105353Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:3 with 3 nodes.
2022-12-01T10:43:19.2105686Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 3 nodes.
2022-12-01T10:43:19.2106018Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 3 nodes.
2022-12-01T10:43:19.2106244Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 0
2022-12-01T10:43:19.2106494Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 2
2022-12-01T10:43:19.2106674Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 1
2022-12-01T10:43:19.2107067Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:4 with 3 nodes.
2022-12-01T10:43:19.2107473Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:4 with 3 nodes.
2022-12-01T10:43:19.2107890Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:4 with 3 nodes.
2022-12-01T10:43:19.2108006Z ok (2.153s)
2022-12-01T10:43:19.2108014Z 
2022-12-01T10:43:19.2108282Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2108404Z Ran 1 test in 2.153s
2022-12-01T10:43:19.2108409Z 
2022-12-01T10:43:19.2108472Z OK
2022-12-01T10:43:19.2108476Z 
2022-12-01T10:43:19.2108562Z Generating XML reports...
2022-12-01T10:43:19.2108912Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103249.xml
2022-12-01T10:43:19.2109147Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.2109425Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2109537Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2109870Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2110067Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2110074Z 
2022-12-01T10:43:19.2110184Z Running tests...
2022-12-01T10:43:19.2110488Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2110769Z   test_all_gather_v_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports all_gather_v (0.002s)
2022-12-01T10:43:19.2110775Z 
2022-12-01T10:43:19.2110971Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2111049Z Ran 1 test in 0.002s
2022-12-01T10:43:19.2111053Z 
2022-12-01T10:43:19.2111127Z OK (skipped=1)
2022-12-01T10:43:19.2111132Z 
2022-12-01T10:43:19.2111204Z Generating XML reports...
2022-12-01T10:43:19.2111552Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103253.xml
2022-12-01T10:43:19.2111785Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.2112063Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2112187Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2112470Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2112605Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2112610Z 
2022-12-01T10:43:19.2112684Z Running tests...
2022-12-01T10:43:19.2112865Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2113119Z   test_all_reduce_coalesced_full_group_max (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30393
2022-12-01T10:43:19.2113344Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30394
2022-12-01T10:43:19.2113531Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 30395
2022-12-01T10:43:19.2113805Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2113930Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2114212Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2114356Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2114537Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:43:19.2114796Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2114922Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2115205Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2115343Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2115520Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:43:19.2115795Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2115918Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2116199Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2116336Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2116501Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:43:19.2116805Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2117106Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2117400Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2117566Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:43:19.2117732Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:43:19.2117894Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:43:19.2118069Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T10:43:19.2118246Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T10:43:19.2118405Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-12-01T10:43:19.2118704Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:43:19.2119000Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:43:19.2119294Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:43:19.2119543Z STAGE:2022-12-01 10:32:55 30393:30393 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2119786Z STAGE:2022-12-01 10:32:55 30395:30395 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2120022Z STAGE:2022-12-01 10:32:55 30394:30394 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2120623Z STAGE:2022-12-01 10:32:55 30394:30394 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 10:32:55 30395:30395 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 10:32:55 30393:30393 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2120655Z 
2022-12-01T10:43:19.2120659Z 
2022-12-01T10:43:19.2120903Z STAGE:2022-12-01 10:32:55 30394:30394 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2121141Z STAGE:2022-12-01 10:32:55 30395:30395 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2121361Z STAGE:2022-12-01 10:32:55 30393:30393 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2121610Z STAGE:2022-12-01 10:32:55 30395:30395 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2122020Z STAGE:2022-12-01 10:32:55 30393:30393 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 10:32:55 30394:30394 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2122028Z 
2022-12-01T10:43:19.2122271Z STAGE:2022-12-01 10:32:55 30395:30395 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2122674Z STAGE:2022-12-01 10:32:55 30393:30393 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-12-01 10:32:55 30394:30394 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2122679Z 
2022-12-01T10:43:19.2122927Z STAGE:2022-12-01 10:32:55 30395:30395 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2123337Z STAGE:2022-12-01 10:32:55 30393:30393 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 10:32:55 30394:30394 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2123343Z 
2022-12-01T10:43:19.2123922Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1578: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-12-01T10:43:19.2124078Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2022-12-01T10:43:19.2124639Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1578: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-12-01T10:43:19.2124787Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2022-12-01T10:43:19.2125342Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1578: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-12-01T10:43:19.2125493Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2022-12-01T10:43:19.2125565Z ok (1.854s)
2022-12-01T10:43:19.2125570Z 
2022-12-01T10:43:19.2125756Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2125833Z Ran 1 test in 1.855s
2022-12-01T10:43:19.2125838Z 
2022-12-01T10:43:19.2125899Z OK
2022-12-01T10:43:19.2125904Z 
2022-12-01T10:43:19.2125988Z Generating XML reports...
2022-12-01T10:43:19.2126328Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103254.xml
2022-12-01T10:43:19.2126560Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.2126834Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2126959Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2127259Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2127426Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2127431Z 
2022-12-01T10:43:19.2127504Z Running tests...
2022-12-01T10:43:19.2127699Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2127950Z   test_all_reduce_coalesced_full_group_min (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30458
2022-12-01T10:43:19.2128109Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30459
2022-12-01T10:43:19.2128265Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 30460
2022-12-01T10:43:19.2128539Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2128668Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2128935Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2129072Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2129253Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:43:19.2129524Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2129647Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2129927Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2130062Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2130241Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:43:19.2130497Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2130622Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2130899Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2131033Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2131208Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:43:19.2131507Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2131805Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2132099Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2132267Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:43:19.2132419Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:43:19.2132581Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:43:19.2132760Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T10:43:19.2132935Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T10:43:19.2133175Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-12-01T10:43:19.2133545Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:43:19.2133878Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:43:19.2134196Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:43:19.2134445Z STAGE:2022-12-01 10:32:59 30458:30458 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2134685Z STAGE:2022-12-01 10:32:59 30459:30459 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2134909Z STAGE:2022-12-01 10:32:59 30460:30460 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2135157Z STAGE:2022-12-01 10:32:59 30459:30459 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2135399Z STAGE:2022-12-01 10:32:59 30460:30460 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2135647Z STAGE:2022-12-01 10:32:59 30458:30458 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2136048Z STAGE:2022-12-01 10:32:59 30459:30459 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-12-01 10:32:59 30460:30460 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2136056Z 
2022-12-01T10:43:19.2136297Z STAGE:2022-12-01 10:32:59 30458:30458 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2136542Z STAGE:2022-12-01 10:32:59 30459:30459 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2136953Z STAGE:2022-12-01 10:32:59 30460:30460 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 10:32:59 30458:30458 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2136959Z 
2022-12-01T10:43:19.2137358Z STAGE:2022-12-01 10:32:59 30458:30458 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-12-01 10:32:59 30459:30459 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2137363Z 
2022-12-01T10:43:19.2137605Z STAGE:2022-12-01 10:32:59 30460:30460 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2137851Z STAGE:2022-12-01 10:32:59 30460:30460 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2138079Z STAGE:2022-12-01 10:32:59 30459:30459 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2138319Z STAGE:2022-12-01 10:32:59 30458:30458 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2138887Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1578: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-12-01T10:43:19.2139040Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2022-12-01T10:43:19.2139594Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1578: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-12-01T10:43:19.2139750Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2022-12-01T10:43:19.2140305Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1578: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-12-01T10:43:19.2140630Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2022-12-01T10:43:19.2140708Z ok (1.955s)
2022-12-01T10:43:19.2140714Z 
2022-12-01T10:43:19.2140914Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2140977Z Ran 1 test in 1.955s
2022-12-01T10:43:19.2140996Z 
2022-12-01T10:43:19.2141116Z OK
2022-12-01T10:43:19.2141121Z 
2022-12-01T10:43:19.2141244Z Generating XML reports...
2022-12-01T10:43:19.2141586Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103257.xml
2022-12-01T10:43:19.2141822Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.2142097Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2142223Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2142504Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2142641Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2142646Z 
2022-12-01T10:43:19.2142705Z Running tests...
2022-12-01T10:43:19.2142901Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2143157Z   test_all_reduce_coalesced_full_group_product (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30523
2022-12-01T10:43:19.2143318Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30524
2022-12-01T10:43:19.2143475Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 30525
2022-12-01T10:43:19.2143751Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2143877Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2144158Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2144293Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2144460Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:43:19.2144732Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2144859Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2145137Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2145272Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2145450Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:43:19.2145721Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2145845Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2146112Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2146247Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2146450Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:43:19.2146812Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2147112Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2147465Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2147650Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:43:19.2147873Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:43:19.2148168Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:43:19.2148339Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T10:43:19.2148547Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T10:43:19.2148731Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-12-01T10:43:19.2149123Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:43:19.2149563Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:43:19.2149923Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:43:19.2150180Z STAGE:2022-12-01 10:33:02 30523:30523 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2150589Z STAGE:2022-12-01 10:33:02 30524:30524 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-12-01 10:33:02 30525:30525 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2150597Z 
2022-12-01T10:43:19.2150848Z STAGE:2022-12-01 10:33:02 30525:30525 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2151099Z STAGE:2022-12-01 10:33:02 30524:30524 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2151330Z STAGE:2022-12-01 10:33:02 30523:30523 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2151573Z STAGE:2022-12-01 10:33:02 30525:30525 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2151812Z STAGE:2022-12-01 10:33:02 30524:30524 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2152047Z STAGE:2022-12-01 10:33:02 30523:30523 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2152300Z STAGE:2022-12-01 10:33:02 30525:30525 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2152547Z STAGE:2022-12-01 10:33:02 30523:30523 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2152788Z STAGE:2022-12-01 10:33:02 30524:30524 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2153029Z STAGE:2022-12-01 10:33:02 30525:30525 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2153267Z STAGE:2022-12-01 10:33:02 30524:30524 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2153497Z STAGE:2022-12-01 10:33:02 30523:30523 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2153746Z STAGE:2022-12-01 10:33:02 30525:30525 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2153987Z STAGE:2022-12-01 10:33:02 30524:30524 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2154229Z STAGE:2022-12-01 10:33:02 30523:30523 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2154802Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1578: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-12-01T10:43:19.2154959Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2022-12-01T10:43:19.2155518Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1578: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-12-01T10:43:19.2155670Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2022-12-01T10:43:19.2156272Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1578: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-12-01T10:43:19.2156447Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2022-12-01T10:43:19.2156517Z ok (1.958s)
2022-12-01T10:43:19.2156522Z 
2022-12-01T10:43:19.2156705Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2156780Z Ran 1 test in 1.958s
2022-12-01T10:43:19.2156785Z 
2022-12-01T10:43:19.2156848Z OK
2022-12-01T10:43:19.2156852Z 
2022-12-01T10:43:19.2156936Z Generating XML reports...
2022-12-01T10:43:19.2157277Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103301.xml
2022-12-01T10:43:19.2157518Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.2157795Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2157923Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2158205Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2158329Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2158334Z 
2022-12-01T10:43:19.2158409Z Running tests...
2022-12-01T10:43:19.2158603Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2158854Z   test_all_reduce_coalesced_full_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30588
2022-12-01T10:43:19.2159012Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30589
2022-12-01T10:43:19.2159171Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 30590
2022-12-01T10:43:19.2159450Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2159575Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2159841Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2159977Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2160158Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:43:19.2160432Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2160558Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2160842Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2160980Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2161163Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:43:19.2161431Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2161541Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2161820Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2161954Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2162132Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:43:19.2162432Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2162760Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2163085Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2163252Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:43:19.2163418Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:43:19.2163569Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:43:19.2163746Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T10:43:19.2163920Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-12-01T10:43:19.2164217Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:43:19.2164393Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T10:43:19.2164690Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:43:19.2164982Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:43:19.2165231Z STAGE:2022-12-01 10:33:06 30588:30588 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2165638Z STAGE:2022-12-01 10:33:06 30590:30590 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-12-01 10:33:06 30589:30589 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2165644Z 
2022-12-01T10:43:19.2165893Z STAGE:2022-12-01 10:33:06 30588:30588 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2166293Z STAGE:2022-12-01 10:33:06 30589:30589 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 10:33:06 30590:30590 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2166314Z 
2022-12-01T10:43:19.2166543Z STAGE:2022-12-01 10:33:06 30588:30588 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2166941Z STAGE:2022-12-01 10:33:06 30589:30589 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-12-01 10:33:06 30590:30590 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2166946Z 
2022-12-01T10:43:19.2167193Z STAGE:2022-12-01 10:33:06 30588:30588 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2167435Z STAGE:2022-12-01 10:33:06 30589:30589 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2167672Z STAGE:2022-12-01 10:33:06 30590:30590 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2167914Z STAGE:2022-12-01 10:33:06 30588:30588 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2168313Z STAGE:2022-12-01 10:33:06 30589:30589 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-12-01 10:33:06 30590:30590 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2168319Z 
2022-12-01T10:43:19.2168567Z STAGE:2022-12-01 10:33:06 30588:30588 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2168810Z STAGE:2022-12-01 10:33:06 30590:30590 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2169051Z STAGE:2022-12-01 10:33:06 30589:30589 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2169627Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1578: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-12-01T10:43:19.2169808Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2022-12-01T10:43:19.2170366Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1578: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-12-01T10:43:19.2170517Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2022-12-01T10:43:19.2171079Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1578: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-12-01T10:43:19.2171227Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2022-12-01T10:43:19.2171299Z ok (1.957s)
2022-12-01T10:43:19.2171304Z 
2022-12-01T10:43:19.2171509Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2171586Z Ran 1 test in 1.957s
2022-12-01T10:43:19.2171591Z 
2022-12-01T10:43:19.2171653Z OK
2022-12-01T10:43:19.2171657Z 
2022-12-01T10:43:19.2171742Z Generating XML reports...
2022-12-01T10:43:19.2172070Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103304.xml
2022-12-01T10:43:19.2172303Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.2172577Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2172705Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2172984Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2173123Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2173131Z 
2022-12-01T10:43:19.2173204Z Running tests...
2022-12-01T10:43:19.2173470Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2173703Z   test_all_reduce_coalesced_group_max (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30653
2022-12-01T10:43:19.2173862Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30654
2022-12-01T10:43:19.2174018Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 30655
2022-12-01T10:43:19.2174291Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2174415Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2174702Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2174840Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2175022Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:43:19.2175292Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2175403Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2175687Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2175823Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2176002Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:43:19.2176274Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2176433Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2176742Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2176876Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2177053Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:43:19.2177339Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2177636Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2177932Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2178101Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:43:19.2178268Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:43:19.2178439Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:43:19.2178616Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T10:43:19.2178792Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T10:43:19.2179019Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-12-01T10:43:19.2179299Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:43:19.2179590Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:43:19.2179842Z STAGE:2022-12-01 10:33:09 30654:30654 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2180087Z STAGE:2022-12-01 10:33:09 30655:30655 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2180645Z STAGE:2022-12-01 10:33:09 30654:30654 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 10:33:09 30655:30655 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2180652Z 
2022-12-01T10:43:19.2181056Z STAGE:2022-12-01 10:33:09 30654:30654 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-12-01 10:33:09 30655:30655 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2181061Z 
2022-12-01T10:43:19.2181310Z STAGE:2022-12-01 10:33:09 30654:30654 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2181555Z STAGE:2022-12-01 10:33:09 30655:30655 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2181852Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:43:19.2182422Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1578: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-12-01T10:43:19.2182574Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2022-12-01T10:43:19.2183110Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1578: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-12-01T10:43:19.2183263Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2022-12-01T10:43:19.2183333Z ok (1.953s)
2022-12-01T10:43:19.2183338Z 
2022-12-01T10:43:19.2183597Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2183707Z Ran 1 test in 1.953s
2022-12-01T10:43:19.2183712Z 
2022-12-01T10:43:19.2183773Z OK
2022-12-01T10:43:19.2183778Z 
2022-12-01T10:43:19.2183863Z Generating XML reports...
2022-12-01T10:43:19.2184210Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103308.xml
2022-12-01T10:43:19.2184445Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.2184707Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2184833Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2185116Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2185257Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2185262Z 
2022-12-01T10:43:19.2185339Z Running tests...
2022-12-01T10:43:19.2185536Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2185788Z   test_all_reduce_coalesced_group_min (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30714
2022-12-01T10:43:19.2185947Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30715
2022-12-01T10:43:19.2186104Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 30716
2022-12-01T10:43:19.2186368Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2186494Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2186775Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2186913Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2187097Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:43:19.2187369Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2187494Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2187777Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2187899Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2188079Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:43:19.2188351Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2188477Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2188754Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2188892Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2189070Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:43:19.2189370Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2189667Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2189945Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2190111Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:43:19.2190321Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:43:19.2190511Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:43:19.2190689Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T10:43:19.2190862Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-12-01T10:43:19.2191033Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T10:43:19.2191326Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:43:19.2191623Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:43:19.2191899Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:43:19.2192151Z STAGE:2022-12-01 10:33:12 30715:30715 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2192400Z STAGE:2022-12-01 10:33:12 30716:30716 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2192650Z STAGE:2022-12-01 10:33:12 30716:30716 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2192893Z STAGE:2022-12-01 10:33:12 30715:30715 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2193131Z STAGE:2022-12-01 10:33:12 30716:30716 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2193374Z STAGE:2022-12-01 10:33:12 30715:30715 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2193622Z STAGE:2022-12-01 10:33:12 30715:30715 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2193865Z STAGE:2022-12-01 10:33:12 30716:30716 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2194432Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1578: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-12-01T10:43:19.2194574Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2022-12-01T10:43:19.2195127Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1578: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-12-01T10:43:19.2195278Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2022-12-01T10:43:19.2195348Z ok (1.853s)
2022-12-01T10:43:19.2195353Z 
2022-12-01T10:43:19.2195551Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2195628Z Ran 1 test in 1.853s
2022-12-01T10:43:19.2195635Z 
2022-12-01T10:43:19.2195697Z OK
2022-12-01T10:43:19.2195702Z 
2022-12-01T10:43:19.2195786Z Generating XML reports...
2022-12-01T10:43:19.2196130Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103311.xml
2022-12-01T10:43:19.2196347Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.2196623Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2196748Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2197027Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2197164Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2197169Z 
2022-12-01T10:43:19.2197270Z Running tests...
2022-12-01T10:43:19.2197467Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2197744Z   test_all_reduce_coalesced_group_product (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30775
2022-12-01T10:43:19.2197905Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30776
2022-12-01T10:43:19.2198048Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 30777
2022-12-01T10:43:19.2198318Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2198441Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2198723Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2198861Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2199040Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:43:19.2199315Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2199440Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2199721Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2199843Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2200021Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:43:19.2200321Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2200594Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2200718Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2201002Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2201138Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2201315Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:43:19.2201596Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2201895Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2202063Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:43:19.2202231Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:43:19.2202406Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T10:43:19.2202569Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:43:19.2202738Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-12-01T10:43:19.2202910Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T10:43:19.2203205Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:43:19.2203484Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:43:19.2203735Z STAGE:2022-12-01 10:33:16 30776:30776 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2204011Z STAGE:2022-12-01 10:33:16 30777:30777 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2204452Z STAGE:2022-12-01 10:33:16 30777:30777 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 10:33:16 30776:30776 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2204457Z 
2022-12-01T10:43:19.2204698Z STAGE:2022-12-01 10:33:16 30776:30776 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2204934Z STAGE:2022-12-01 10:33:16 30777:30777 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2205183Z STAGE:2022-12-01 10:33:16 30777:30777 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2205427Z STAGE:2022-12-01 10:33:16 30776:30776 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2205726Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:43:19.2206292Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1578: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-12-01T10:43:19.2206447Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2022-12-01T10:43:19.2206986Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1578: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-12-01T10:43:19.2207137Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2022-12-01T10:43:19.2207205Z ok (1.955s)
2022-12-01T10:43:19.2207210Z 
2022-12-01T10:43:19.2207410Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2207489Z Ran 1 test in 1.955s
2022-12-01T10:43:19.2207494Z 
2022-12-01T10:43:19.2207556Z OK
2022-12-01T10:43:19.2207561Z 
2022-12-01T10:43:19.2207645Z Generating XML reports...
2022-12-01T10:43:19.2207990Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103314.xml
2022-12-01T10:43:19.2208224Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.2208485Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2208610Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2208889Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2209029Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2209034Z 
2022-12-01T10:43:19.2209107Z Running tests...
2022-12-01T10:43:19.2209304Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2209554Z   test_all_reduce_coalesced_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30836
2022-12-01T10:43:19.2209712Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30837
2022-12-01T10:43:19.2209870Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 30838
2022-12-01T10:43:19.2210130Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2210253Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2210535Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2210673Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2210882Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:43:19.2211215Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2211486Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2211608Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2211877Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2212012Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2212190Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:43:19.2212462Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2212587Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2212871Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2213007Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2213185Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:43:19.2213573Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2213859Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2214029Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:43:19.2214192Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:43:19.2214370Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T10:43:19.2214546Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-12-01T10:43:19.2214709Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:43:19.2214881Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T10:43:19.2215173Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:43:19.2215467Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:43:19.2215741Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:43:19.2215994Z STAGE:2022-12-01 10:33:19 30838:30838 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2216235Z STAGE:2022-12-01 10:33:19 30837:30837 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2216656Z STAGE:2022-12-01 10:33:19 30837:30837 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 10:33:19 30838:30838 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2216663Z 
2022-12-01T10:43:19.2216909Z STAGE:2022-12-01 10:33:19 30838:30838 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2217152Z STAGE:2022-12-01 10:33:19 30837:30837 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2217399Z STAGE:2022-12-01 10:33:19 30837:30837 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2217644Z STAGE:2022-12-01 10:33:19 30838:30838 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2218250Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1578: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-12-01T10:43:19.2218430Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2022-12-01T10:43:19.2218987Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1578: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-12-01T10:43:19.2219125Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2022-12-01T10:43:19.2219194Z ok (1.952s)
2022-12-01T10:43:19.2219199Z 
2022-12-01T10:43:19.2219396Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2219474Z Ran 1 test in 1.952s
2022-12-01T10:43:19.2219480Z 
2022-12-01T10:43:19.2219545Z OK
2022-12-01T10:43:19.2219549Z 
2022-12-01T10:43:19.2219632Z Generating XML reports...
2022-12-01T10:43:19.2219976Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103318.xml
2022-12-01T10:43:19.2220209Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.2220648Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2220760Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2221046Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2221182Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2221187Z 
2022-12-01T10:43:19.2221264Z Running tests...
2022-12-01T10:43:19.2221462Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2221706Z   test_all_reduce_coalesced_max (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30897
2022-12-01T10:43:19.2221866Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30898
2022-12-01T10:43:19.2222022Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 30899
2022-12-01T10:43:19.2222283Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2222408Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2222691Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2222827Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2223005Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:43:19.2223278Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2223404Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2223684Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2223824Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2223988Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:43:19.2224259Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2224383Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2224661Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2224849Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2225070Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:43:19.2225373Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2225671Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2225963Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2226117Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:43:19.2226283Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:43:19.2226535Z STAGE:2022-12-01 10:33:23 30898:30898 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2226778Z STAGE:2022-12-01 10:33:23 30899:30899 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2226944Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:43:19.2227182Z STAGE:2022-12-01 10:33:23 30897:30897 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2227430Z STAGE:2022-12-01 10:33:23 30897:30897 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2227838Z STAGE:2022-12-01 10:33:23 30898:30898 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 10:33:23 30899:30899 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2227844Z 
2022-12-01T10:43:19.2228085Z STAGE:2022-12-01 10:33:23 30897:30897 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2228331Z STAGE:2022-12-01 10:33:23 30898:30898 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2228559Z STAGE:2022-12-01 10:33:23 30899:30899 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2228972Z STAGE:2022-12-01 10:33:23 30898:30898 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 10:33:23 30899:30899 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2228978Z 
2022-12-01T10:43:19.2229227Z STAGE:2022-12-01 10:33:23 30897:30897 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2229467Z STAGE:2022-12-01 10:33:23 30898:30898 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2229704Z STAGE:2022-12-01 10:33:23 30899:30899 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2229941Z STAGE:2022-12-01 10:33:23 30897:30897 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2230192Z STAGE:2022-12-01 10:33:23 30899:30899 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2230433Z STAGE:2022-12-01 10:33:23 30898:30898 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2230684Z STAGE:2022-12-01 10:33:23 30897:30897 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2231248Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1578: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-12-01T10:43:19.2231390Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2022-12-01T10:43:19.2231939Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1578: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-12-01T10:43:19.2232119Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2022-12-01T10:43:19.2232702Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1578: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-12-01T10:43:19.2232850Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2022-12-01T10:43:19.2232922Z ok (1.952s)
2022-12-01T10:43:19.2232927Z 
2022-12-01T10:43:19.2233125Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2233202Z Ran 1 test in 1.952s
2022-12-01T10:43:19.2233207Z 
2022-12-01T10:43:19.2233269Z OK
2022-12-01T10:43:19.2233273Z 
2022-12-01T10:43:19.2233359Z Generating XML reports...
2022-12-01T10:43:19.2233688Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103321.xml
2022-12-01T10:43:19.2233925Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.2234202Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2234329Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2234610Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2234748Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2234753Z 
2022-12-01T10:43:19.2234827Z Running tests...
2022-12-01T10:43:19.2235022Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2235288Z   test_all_reduce_coalesced_max_complex_unsupported (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30953
2022-12-01T10:43:19.2235435Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30954
2022-12-01T10:43:19.2235595Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 30955
2022-12-01T10:43:19.2235869Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2235993Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2236276Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2236414Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2236594Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:43:19.2236866Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2236979Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2237260Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2237397Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2237576Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:43:19.2237875Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2238149Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2238273Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2238551Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2238731Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2238893Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:43:19.2239221Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2239517Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2239684Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:43:19.2239848Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:43:19.2240014Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:43:19.2240583Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1578: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-12-01T10:43:19.2240807Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2022-12-01T10:43:19.2241369Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1578: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-12-01T10:43:19.2241516Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2022-12-01T10:43:19.2242064Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1578: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-12-01T10:43:19.2242203Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2022-12-01T10:43:19.2242273Z ok (1.752s)
2022-12-01T10:43:19.2242281Z 
2022-12-01T10:43:19.2242476Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2242554Z Ran 1 test in 1.752s
2022-12-01T10:43:19.2242559Z 
2022-12-01T10:43:19.2242620Z OK
2022-12-01T10:43:19.2242624Z 
2022-12-01T10:43:19.2242709Z Generating XML reports...
2022-12-01T10:43:19.2243049Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103324.xml
2022-12-01T10:43:19.2243282Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.2243542Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2243667Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2243950Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2244088Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2244094Z 
2022-12-01T10:43:19.2244167Z Running tests...
2022-12-01T10:43:19.2244363Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2244607Z   test_all_reduce_coalesced_min (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31006
2022-12-01T10:43:19.2244766Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31007
2022-12-01T10:43:19.2244922Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 31008
2022-12-01T10:43:19.2245182Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2245307Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2245621Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2245807Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2245986Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:43:19.2246262Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2246387Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2246666Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2246801Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2246965Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:43:19.2247237Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2247365Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2247670Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2247811Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2247988Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:43:19.2248290Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2248586Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2248876Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2249031Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:43:19.2249199Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:43:19.2249450Z STAGE:2022-12-01 10:33:29 31008:31008 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2249692Z STAGE:2022-12-01 10:33:29 31007:31007 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2249853Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:43:19.2250094Z STAGE:2022-12-01 10:33:29 31006:31006 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2250343Z STAGE:2022-12-01 10:33:29 31006:31006 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2250588Z STAGE:2022-12-01 10:33:29 31008:31008 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2250816Z STAGE:2022-12-01 10:33:29 31007:31007 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2251057Z STAGE:2022-12-01 10:33:29 31006:31006 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2251466Z STAGE:2022-12-01 10:33:29 31008:31008 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-12-01 10:33:29 31007:31007 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2251472Z 
2022-12-01T10:43:19.2251721Z STAGE:2022-12-01 10:33:29 31008:31008 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2252129Z STAGE:2022-12-01 10:33:29 31006:31006 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 10:33:29 31007:31007 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2252135Z 
2022-12-01T10:43:19.2252374Z STAGE:2022-12-01 10:33:29 31008:31008 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2252809Z STAGE:2022-12-01 10:33:29 31006:31006 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-12-01 10:33:29 31007:31007 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2252840Z 
2022-12-01T10:43:19.2253086Z STAGE:2022-12-01 10:33:29 31008:31008 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2253571Z STAGE:2022-12-01 10:33:29 31006:31006 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 10:33:29 31007:31007 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2253577Z 
2022-12-01T10:43:19.2254142Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1578: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-12-01T10:43:19.2254299Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2022-12-01T10:43:19.2254865Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1578: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-12-01T10:43:19.2255016Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2022-12-01T10:43:19.2255569Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1578: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-12-01T10:43:19.2255719Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2022-12-01T10:43:19.2255774Z ok (1.960s)
2022-12-01T10:43:19.2255779Z 
2022-12-01T10:43:19.2255976Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2256055Z Ran 1 test in 1.960s
2022-12-01T10:43:19.2256060Z 
2022-12-01T10:43:19.2256126Z OK
2022-12-01T10:43:19.2256131Z 
2022-12-01T10:43:19.2256216Z Generating XML reports...
2022-12-01T10:43:19.2256556Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103328.xml
2022-12-01T10:43:19.2256788Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.2257062Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2257174Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2257457Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2257595Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2257601Z 
2022-12-01T10:43:19.2257677Z Running tests...
2022-12-01T10:43:19.2257872Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2258124Z   test_all_reduce_coalesced_product (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31062
2022-12-01T10:43:19.2258286Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31063
2022-12-01T10:43:19.2258442Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 31064
2022-12-01T10:43:19.2258715Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2258826Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2259106Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2259242Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2259460Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:43:19.2259806Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2259930Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2260212Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2260475Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2260656Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:43:19.2260931Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2261056Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2261340Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2261478Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2261655Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:43:19.2261956Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2262258Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2262548Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2262701Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:43:19.2262865Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:43:19.2263121Z STAGE:2022-12-01 10:33:33 31064:31064 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2263365Z STAGE:2022-12-01 10:33:33 31063:31063 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2263529Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:43:19.2263770Z STAGE:2022-12-01 10:33:33 31062:31062 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2264019Z STAGE:2022-12-01 10:33:33 31062:31062 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2264264Z STAGE:2022-12-01 10:33:33 31063:31063 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2264507Z STAGE:2022-12-01 10:33:33 31064:31064 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2264732Z STAGE:2022-12-01 10:33:33 31062:31062 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2264979Z STAGE:2022-12-01 10:33:33 31064:31064 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2265219Z STAGE:2022-12-01 10:33:33 31063:31063 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2265630Z STAGE:2022-12-01 10:33:33 31063:31063 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 10:33:33 31064:31064 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2265636Z 
2022-12-01T10:43:19.2265883Z STAGE:2022-12-01 10:33:33 31062:31062 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2266124Z STAGE:2022-12-01 10:33:33 31062:31062 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2266361Z STAGE:2022-12-01 10:33:33 31063:31063 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2266597Z STAGE:2022-12-01 10:33:33 31064:31064 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2266896Z STAGE:2022-12-01 10:33:33 31063:31063 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2267170Z STAGE:2022-12-01 10:33:33 31062:31062 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2267401Z STAGE:2022-12-01 10:33:33 31064:31064 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2267965Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1578: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-12-01T10:43:19.2268116Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2022-12-01T10:43:19.2268669Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1578: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-12-01T10:43:19.2268818Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2022-12-01T10:43:19.2269379Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1578: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-12-01T10:43:19.2269525Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2022-12-01T10:43:19.2269594Z ok (1.958s)
2022-12-01T10:43:19.2269599Z 
2022-12-01T10:43:19.2269798Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2269874Z Ran 1 test in 1.958s
2022-12-01T10:43:19.2269879Z 
2022-12-01T10:43:19.2269926Z OK
2022-12-01T10:43:19.2269945Z 
2022-12-01T10:43:19.2270015Z Generating XML reports...
2022-12-01T10:43:19.2270360Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103331.xml
2022-12-01T10:43:19.2270598Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.2270877Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2271004Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2271285Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2271422Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2271427Z 
2022-12-01T10:43:19.2271501Z Running tests...
2022-12-01T10:43:19.2271682Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2271924Z   test_all_reduce_coalesced_sum (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31118
2022-12-01T10:43:19.2272083Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31119
2022-12-01T10:43:19.2272240Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 31120
2022-12-01T10:43:19.2272513Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2272643Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2272927Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2273066Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2273232Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:43:19.2273536Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2273661Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2273971Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2274106Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2274287Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:43:19.2274590Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2274859Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2274983Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2275268Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2275405Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2275582Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:43:19.2275881Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2276178Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2276345Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:43:19.2276512Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:43:19.2276677Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:43:19.2276927Z STAGE:2022-12-01 10:33:36 31118:31118 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2277164Z STAGE:2022-12-01 10:33:36 31120:31120 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2277405Z STAGE:2022-12-01 10:33:36 31119:31119 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2277818Z STAGE:2022-12-01 10:33:36 31119:31119 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 10:33:36 31120:31120 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2278068Z STAGE:2022-12-01 10:33:36 31118:31118 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2278073Z 
2022-12-01T10:43:19.2278624Z STAGE:2022-12-01 10:33:36 31120:31120 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-12-01 10:33:36 31119:31119 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-12-01 10:33:36 31118:31118 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2278630Z 
2022-12-01T10:43:19.2278634Z 
2022-12-01T10:43:19.2278884Z STAGE:2022-12-01 10:33:36 31119:31119 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2279132Z STAGE:2022-12-01 10:33:36 31118:31118 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2279380Z STAGE:2022-12-01 10:33:36 31120:31120 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2279620Z STAGE:2022-12-01 10:33:36 31119:31119 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2280015Z STAGE:2022-12-01 10:33:36 31120:31120 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-12-01 10:33:36 31118:31118 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2280021Z 
2022-12-01T10:43:19.2280267Z STAGE:2022-12-01 10:33:36 31119:31119 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2280709Z STAGE:2022-12-01 10:33:36 31120:31120 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 10:33:36 31118:31118 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2280745Z 
2022-12-01T10:43:19.2281313Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1578: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-12-01T10:43:19.2281465Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2022-12-01T10:43:19.2282006Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1578: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-12-01T10:43:19.2282158Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2022-12-01T10:43:19.2282718Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1578: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-12-01T10:43:19.2282871Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2022-12-01T10:43:19.2282943Z ok (1.751s)
2022-12-01T10:43:19.2282948Z 
2022-12-01T10:43:19.2283147Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2283227Z Ran 1 test in 1.751s
2022-12-01T10:43:19.2283233Z 
2022-12-01T10:43:19.2283296Z OK
2022-12-01T10:43:19.2283301Z 
2022-12-01T10:43:19.2283389Z Generating XML reports...
2022-12-01T10:43:19.2283718Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103335.xml
2022-12-01T10:43:19.2283958Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.2284236Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2284367Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2284650Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2284789Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2284795Z 
2022-12-01T10:43:19.2284872Z Running tests...
2022-12-01T10:43:19.2285069Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2285324Z   test_all_reduce_complex_unsupported_ops (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31174
2022-12-01T10:43:19.2285470Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31175
2022-12-01T10:43:19.2285630Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 31176
2022-12-01T10:43:19.2285909Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2286036Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2286317Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2286454Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2286635Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:43:19.2286907Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2287033Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2287343Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2287480Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2287688Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:43:19.2287965Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2288088Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2288367Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2288502Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2288679Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:43:19.2288965Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2289266Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2289564Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2289731Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:43:19.2289896Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:43:19.2290061Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:43:19.2290131Z ok (1.956s)
2022-12-01T10:43:19.2290137Z 
2022-12-01T10:43:19.2290336Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2290412Z Ran 1 test in 1.956s
2022-12-01T10:43:19.2290417Z 
2022-12-01T10:43:19.2290464Z OK
2022-12-01T10:43:19.2290483Z 
2022-12-01T10:43:19.2290557Z Generating XML reports...
2022-12-01T10:43:19.2290898Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103338.xml
2022-12-01T10:43:19.2291133Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.2291406Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2291531Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2291812Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2291949Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2291954Z 
2022-12-01T10:43:19.2292028Z Running tests...
2022-12-01T10:43:19.2292209Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2292448Z   test_all_reduce_full_group_max (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31227
2022-12-01T10:43:19.2292609Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31228
2022-12-01T10:43:19.2292764Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 31229
2022-12-01T10:43:19.2293038Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2293162Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2293525Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2293661Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2293824Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:43:19.2294141Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2294298Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2294581Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2294718Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2294897Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:43:19.2295168Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2295295Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2295573Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2295692Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2295871Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:43:19.2296174Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2296471Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2296762Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2296931Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:43:19.2297096Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:43:19.2297260Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:43:19.2297441Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T10:43:19.2297604Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T10:43:19.2297775Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-12-01T10:43:19.2298070Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:43:19.2298366Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:43:19.2298655Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:43:19.2298906Z STAGE:2022-12-01 10:33:43 31229:31229 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2299150Z STAGE:2022-12-01 10:33:43 31227:31227 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2299388Z STAGE:2022-12-01 10:33:43 31228:31228 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2299963Z STAGE:2022-12-01 10:33:43 31227:31227 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 10:33:43 31228:31228 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 10:33:43 31229:31229 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2299969Z 
2022-12-01T10:43:19.2299973Z 
2022-12-01T10:43:19.2300215Z STAGE:2022-12-01 10:33:43 31228:31228 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2300752Z STAGE:2022-12-01 10:33:43 31229:31229 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-12-01 10:33:43 31227:31227 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2300758Z 
2022-12-01T10:43:19.2301065Z STAGE:2022-12-01 10:33:43 31227:31227 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2301298Z STAGE:2022-12-01 10:33:43 31228:31228 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2301582Z STAGE:2022-12-01 10:33:43 31229:31229 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2301827Z STAGE:2022-12-01 10:33:43 31228:31228 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2302067Z STAGE:2022-12-01 10:33:43 31227:31227 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2302306Z STAGE:2022-12-01 10:33:43 31229:31229 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2302552Z STAGE:2022-12-01 10:33:43 31228:31228 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2302966Z STAGE:2022-12-01 10:33:43 31229:31229 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 10:33:43 31227:31227 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2302973Z 
2022-12-01T10:43:19.2303042Z ok (1.955s)
2022-12-01T10:43:19.2303049Z 
2022-12-01T10:43:19.2303249Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2303311Z Ran 1 test in 1.956s
2022-12-01T10:43:19.2303316Z 
2022-12-01T10:43:19.2303377Z OK
2022-12-01T10:43:19.2303383Z 
2022-12-01T10:43:19.2303468Z Generating XML reports...
2022-12-01T10:43:19.2303812Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103341.xml
2022-12-01T10:43:19.2304044Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.2304317Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2304444Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2304728Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2304867Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2304872Z 
2022-12-01T10:43:19.2304933Z Running tests...
2022-12-01T10:43:19.2305129Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2305372Z   test_all_reduce_full_group_min (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31292
2022-12-01T10:43:19.2305531Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31293
2022-12-01T10:43:19.2305687Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 31294
2022-12-01T10:43:19.2305961Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2306086Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2306370Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2306495Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2306675Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:43:19.2306946Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2307073Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2307356Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2307491Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2307669Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:43:19.2307976Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2308130Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2308398Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2308533Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2308711Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:43:19.2309014Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2309311Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2309606Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2309775Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:43:19.2309943Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:43:19.2310106Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:43:19.2310268Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-12-01T10:43:19.2310443Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T10:43:19.2310615Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T10:43:19.2310908Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:43:19.2311204Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:43:19.2311499Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:43:19.2311754Z STAGE:2022-12-01 10:33:46 31292:31292 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2312162Z STAGE:2022-12-01 10:33:46 31293:31293 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-12-01 10:33:46 31294:31294 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2312169Z 
2022-12-01T10:43:19.2312417Z STAGE:2022-12-01 10:33:46 31294:31294 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2312663Z STAGE:2022-12-01 10:33:46 31292:31292 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2312892Z STAGE:2022-12-01 10:33:46 31293:31293 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2313138Z STAGE:2022-12-01 10:33:46 31294:31294 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2313380Z STAGE:2022-12-01 10:33:46 31293:31293 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2313618Z STAGE:2022-12-01 10:33:46 31292:31292 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2313867Z STAGE:2022-12-01 10:33:46 31294:31294 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2314278Z STAGE:2022-12-01 10:33:46 31293:31293 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 10:33:46 31292:31292 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2314285Z 
2022-12-01T10:43:19.2314527Z STAGE:2022-12-01 10:33:46 31294:31294 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2314923Z STAGE:2022-12-01 10:33:46 31292:31292 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-12-01 10:33:46 31293:31293 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2314957Z 
2022-12-01T10:43:19.2315204Z STAGE:2022-12-01 10:33:46 31294:31294 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2315473Z STAGE:2022-12-01 10:33:46 31293:31293 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2315698Z STAGE:2022-12-01 10:33:46 31292:31292 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2315769Z ok (1.954s)
2022-12-01T10:43:19.2315774Z 
2022-12-01T10:43:19.2315972Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2316049Z Ran 1 test in 1.954s
2022-12-01T10:43:19.2316055Z 
2022-12-01T10:43:19.2316118Z OK
2022-12-01T10:43:19.2316122Z 
2022-12-01T10:43:19.2316208Z Generating XML reports...
2022-12-01T10:43:19.2316551Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103345.xml
2022-12-01T10:43:19.2316787Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.2317062Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2317176Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2317464Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2317607Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2317612Z 
2022-12-01T10:43:19.2317686Z Running tests...
2022-12-01T10:43:19.2317884Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2318129Z   test_all_reduce_full_group_product (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31357
2022-12-01T10:43:19.2318287Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31358
2022-12-01T10:43:19.2318448Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 31359
2022-12-01T10:43:19.2318710Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2318835Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2319116Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2319254Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2319432Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:43:19.2319702Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2319826Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2320107Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2320245Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2320409Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:43:19.2320683Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2320811Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2321094Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2321231Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2321409Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:43:19.2321738Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2322041Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2322362Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2322515Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:43:19.2322676Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:43:19.2322841Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:43:19.2323019Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T10:43:19.2323193Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-12-01T10:43:19.2323367Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T10:43:19.2323662Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:43:19.2323955Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:43:19.2324243Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:43:19.2324480Z STAGE:2022-12-01 10:33:50 31357:31357 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2324727Z STAGE:2022-12-01 10:33:50 31359:31359 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2324968Z STAGE:2022-12-01 10:33:50 31358:31358 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2325219Z STAGE:2022-12-01 10:33:50 31357:31357 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2325462Z STAGE:2022-12-01 10:33:50 31358:31358 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2325704Z STAGE:2022-12-01 10:33:50 31359:31359 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2326258Z STAGE:2022-12-01 10:33:50 31359:31359 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-12-01 10:33:50 31358:31358 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-12-01 10:33:50 31357:31357 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2326265Z 
2022-12-01T10:43:19.2326269Z 
2022-12-01T10:43:19.2326510Z STAGE:2022-12-01 10:33:50 31358:31358 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2326918Z STAGE:2022-12-01 10:33:50 31357:31357 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 10:33:50 31359:31359 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2326926Z 
2022-12-01T10:43:19.2327169Z STAGE:2022-12-01 10:33:50 31358:31358 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2327410Z STAGE:2022-12-01 10:33:50 31359:31359 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2327630Z STAGE:2022-12-01 10:33:50 31357:31357 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2327879Z STAGE:2022-12-01 10:33:50 31358:31358 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2328289Z STAGE:2022-12-01 10:33:50 31359:31359 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 10:33:50 31357:31357 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2328295Z 
2022-12-01T10:43:19.2328364Z ok (2.054s)
2022-12-01T10:43:19.2328369Z 
2022-12-01T10:43:19.2328567Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2328646Z Ran 1 test in 2.054s
2022-12-01T10:43:19.2328684Z 
2022-12-01T10:43:19.2328748Z OK
2022-12-01T10:43:19.2328781Z 
2022-12-01T10:43:19.2328868Z Generating XML reports...
2022-12-01T10:43:19.2329212Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103348.xml
2022-12-01T10:43:19.2329445Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.2329708Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2329833Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2330112Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2330249Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2330254Z 
2022-12-01T10:43:19.2330328Z Running tests...
2022-12-01T10:43:19.2330526Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2330770Z   test_all_reduce_full_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31422
2022-12-01T10:43:19.2330928Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31423
2022-12-01T10:43:19.2331071Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 31424
2022-12-01T10:43:19.2331346Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2331470Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2331755Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2331892Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2332072Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:43:19.2332349Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2332473Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2332752Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2332874Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2333054Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:43:19.2333395Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2333522Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2333809Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2333946Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2334126Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:43:19.2334425Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2334723Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2335003Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2335172Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:43:19.2335339Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:43:19.2335548Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:43:19.2335753Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T10:43:19.2335928Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-12-01T10:43:19.2336102Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T10:43:19.2336398Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:43:19.2336692Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:43:19.2336967Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:43:19.2337379Z STAGE:2022-12-01 10:33:53 31422:31422 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-12-01 10:33:53 31424:31424 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2337387Z 
2022-12-01T10:43:19.2337631Z STAGE:2022-12-01 10:33:53 31423:31423 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2337878Z STAGE:2022-12-01 10:33:53 31423:31423 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2338123Z STAGE:2022-12-01 10:33:53 31422:31422 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2338372Z STAGE:2022-12-01 10:33:53 31424:31424 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2338615Z STAGE:2022-12-01 10:33:53 31423:31423 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2339017Z STAGE:2022-12-01 10:33:53 31424:31424 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-12-01 10:33:53 31422:31422 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2339025Z 
2022-12-01T10:43:19.2339272Z STAGE:2022-12-01 10:33:53 31423:31423 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2339682Z STAGE:2022-12-01 10:33:53 31422:31422 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 10:33:53 31424:31424 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2339688Z 
2022-12-01T10:43:19.2339929Z STAGE:2022-12-01 10:33:53 31423:31423 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2340311Z STAGE:2022-12-01 10:33:53 31424:31424 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-12-01 10:33:53 31422:31422 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2340332Z 
2022-12-01T10:43:19.2340737Z STAGE:2022-12-01 10:33:53 31423:31423 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2341152Z STAGE:2022-12-01 10:33:53 31422:31422 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 10:33:53 31424:31424 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2341160Z 
2022-12-01T10:43:19.2341236Z ok (2.053s)
2022-12-01T10:43:19.2341241Z 
2022-12-01T10:43:19.2341439Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2341532Z Ran 1 test in 2.054s
2022-12-01T10:43:19.2341537Z 
2022-12-01T10:43:19.2341597Z OK
2022-12-01T10:43:19.2341602Z 
2022-12-01T10:43:19.2341689Z Generating XML reports...
2022-12-01T10:43:19.2342031Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103351.xml
2022-12-01T10:43:19.2342263Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.2342523Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2342707Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2342990Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2343163Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2343168Z 
2022-12-01T10:43:19.2343245Z Running tests...
2022-12-01T10:43:19.2343441Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2343680Z   test_all_reduce_group_max (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31487
2022-12-01T10:43:19.2343841Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31488
2022-12-01T10:43:19.2343999Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 31489
2022-12-01T10:43:19.2344258Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2344382Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2344668Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2344806Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2344984Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:43:19.2345256Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2345381Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2345660Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2345781Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2345962Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:43:19.2346233Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2346360Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2346639Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2346775Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2346950Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:43:19.2347250Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2347548Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2347828Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2347999Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:43:19.2348160Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:43:19.2348326Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:43:19.2348502Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T10:43:19.2348679Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-12-01T10:43:19.2348855Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T10:43:19.2349148Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:43:19.2349472Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:43:19.2349774Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:43:19.2350025Z STAGE:2022-12-01 10:33:57 31488:31488 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2350273Z STAGE:2022-12-01 10:33:57 31489:31489 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2350524Z STAGE:2022-12-01 10:33:57 31489:31489 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2350771Z STAGE:2022-12-01 10:33:57 31488:31488 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2351013Z STAGE:2022-12-01 10:33:57 31489:31489 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2351251Z STAGE:2022-12-01 10:33:57 31488:31488 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2351499Z STAGE:2022-12-01 10:33:57 31489:31489 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2351743Z STAGE:2022-12-01 10:33:57 31488:31488 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2351798Z ok (1.955s)
2022-12-01T10:43:19.2351803Z 
2022-12-01T10:43:19.2352000Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2352077Z Ran 1 test in 1.956s
2022-12-01T10:43:19.2352082Z 
2022-12-01T10:43:19.2352145Z OK
2022-12-01T10:43:19.2352149Z 
2022-12-01T10:43:19.2352235Z Generating XML reports...
2022-12-01T10:43:19.2352576Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103355.xml
2022-12-01T10:43:19.2352810Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.2353087Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2353212Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2353485Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2353622Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2353627Z 
2022-12-01T10:43:19.2353701Z Running tests...
2022-12-01T10:43:19.2353898Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2354134Z   test_all_reduce_group_min (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31548
2022-12-01T10:43:19.2354292Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31549
2022-12-01T10:43:19.2354448Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 31550
2022-12-01T10:43:19.2354723Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2354836Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2355119Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2355253Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2355430Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:43:19.2355702Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2355829Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2356110Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2356245Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2356481Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:43:19.2356763Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2356885Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2357162Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2357298Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2357475Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:43:19.2357772Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2358071Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2358366Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2358534Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:43:19.2358684Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:43:19.2358860Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T10:43:19.2359033Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-12-01T10:43:19.2359197Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:43:19.2359368Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T10:43:19.2359667Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:43:19.2359963Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:43:19.2360257Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:43:19.2360665Z STAGE:2022-12-01 10:34:00 31550:31550 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-12-01 10:34:00 31549:31549 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2360672Z 
2022-12-01T10:43:19.2361081Z STAGE:2022-12-01 10:34:00 31549:31549 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 10:34:00 31550:31550 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2361087Z 
2022-12-01T10:43:19.2361487Z STAGE:2022-12-01 10:34:00 31549:31549 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-12-01 10:34:00 31550:31550 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2361494Z 
2022-12-01T10:43:19.2361728Z STAGE:2022-12-01 10:34:00 31549:31549 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2361972Z STAGE:2022-12-01 10:34:00 31550:31550 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2362043Z ok (1.953s)
2022-12-01T10:43:19.2362048Z 
2022-12-01T10:43:19.2362247Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2362324Z Ran 1 test in 1.953s
2022-12-01T10:43:19.2362328Z 
2022-12-01T10:43:19.2362390Z OK
2022-12-01T10:43:19.2362394Z 
2022-12-01T10:43:19.2362479Z Generating XML reports...
2022-12-01T10:43:19.2362823Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103358.xml
2022-12-01T10:43:19.2363054Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.2363359Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2363514Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2363800Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2363938Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2363943Z 
2022-12-01T10:43:19.2364016Z Running tests...
2022-12-01T10:43:19.2364217Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2364460Z   test_all_reduce_group_product (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31609
2022-12-01T10:43:19.2364618Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31610
2022-12-01T10:43:19.2364764Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 31611
2022-12-01T10:43:19.2365036Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2365163Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2365446Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2365583Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2365760Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:43:19.2366032Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2366156Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2366436Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2366588Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2366767Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:43:19.2367065Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2367337Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2367461Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2367743Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2367884Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2368060Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:43:19.2368356Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2368642Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2368807Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:43:19.2368970Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:43:19.2369147Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T10:43:19.2369308Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:43:19.2369478Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T10:43:19.2369683Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-12-01T10:43:19.2369978Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:43:19.2370301Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:43:19.2370535Z STAGE:2022-12-01 10:34:03 31610:31610 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2370783Z STAGE:2022-12-01 10:34:03 31611:31611 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2371199Z STAGE:2022-12-01 10:34:03 31611:31611 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 10:34:03 31610:31610 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2371207Z 
2022-12-01T10:43:19.2371452Z STAGE:2022-12-01 10:34:03 31610:31610 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2371696Z STAGE:2022-12-01 10:34:03 31611:31611 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2371949Z STAGE:2022-12-01 10:34:03 31610:31610 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2372192Z STAGE:2022-12-01 10:34:03 31611:31611 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2372488Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:43:19.2372559Z ok (1.957s)
2022-12-01T10:43:19.2372565Z 
2022-12-01T10:43:19.2372764Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2372827Z Ran 1 test in 1.958s
2022-12-01T10:43:19.2372833Z 
2022-12-01T10:43:19.2372893Z OK
2022-12-01T10:43:19.2372898Z 
2022-12-01T10:43:19.2372983Z Generating XML reports...
2022-12-01T10:43:19.2373398Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103402.xml
2022-12-01T10:43:19.2373642Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.2373921Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2374047Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2374328Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2374451Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2374471Z 
2022-12-01T10:43:19.2374530Z Running tests...
2022-12-01T10:43:19.2374728Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2374963Z   test_all_reduce_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31670
2022-12-01T10:43:19.2375126Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31671
2022-12-01T10:43:19.2375282Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 31672
2022-12-01T10:43:19.2375561Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2375690Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2375974Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2376096Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2376275Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:43:19.2376548Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2376675Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2376997Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2377161Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2377341Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:43:19.2377613Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2377722Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2378002Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2378138Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2378316Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:43:19.2378618Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2378917Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2379209Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2379379Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:43:19.2379547Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:43:19.2379695Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:43:19.2379874Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T10:43:19.2380050Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T10:43:19.2380224Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-12-01T10:43:19.2380646Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:43:19.2380942Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:43:19.2381193Z STAGE:2022-12-01 10:34:07 31671:31671 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2381441Z STAGE:2022-12-01 10:34:07 31672:31672 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2381854Z STAGE:2022-12-01 10:34:07 31672:31672 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 10:34:07 31671:31671 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2381860Z 
2022-12-01T10:43:19.2382261Z STAGE:2022-12-01 10:34:07 31672:31672 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-12-01 10:34:07 31671:31671 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2382268Z 
2022-12-01T10:43:19.2382519Z STAGE:2022-12-01 10:34:07 31672:31672 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2382750Z STAGE:2022-12-01 10:34:07 31671:31671 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2383047Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:43:19.2383117Z ok (1.952s)
2022-12-01T10:43:19.2383122Z 
2022-12-01T10:43:19.2383320Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2383400Z Ran 1 test in 1.952s
2022-12-01T10:43:19.2383406Z 
2022-12-01T10:43:19.2383470Z OK
2022-12-01T10:43:19.2383474Z 
2022-12-01T10:43:19.2383560Z Generating XML reports...
2022-12-01T10:43:19.2383970Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103405.xml
2022-12-01T10:43:19.2384225Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.2384500Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2384625Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2384906Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2385045Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2385050Z 
2022-12-01T10:43:19.2385126Z Running tests...
2022-12-01T10:43:19.2385322Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2385558Z   test_all_reduce_max (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31731
2022-12-01T10:43:19.2385715Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31732
2022-12-01T10:43:19.2385859Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 31733
2022-12-01T10:43:19.2386137Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2386261Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2386544Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2386681Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2386861Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:43:19.2387134Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2387262Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2387545Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2387667Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2387846Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:43:19.2388115Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2388241Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2388521Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2388655Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2388835Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:43:19.2389136Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2389420Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2389711Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2389878Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:43:19.2390044Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:43:19.2390295Z STAGE:2022-12-01 10:34:10 31733:31733 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2390539Z STAGE:2022-12-01 10:34:10 31732:31732 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2390735Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:43:19.2391000Z STAGE:2022-12-01 10:34:10 31731:31731 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2391248Z STAGE:2022-12-01 10:34:10 31731:31731 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2391477Z STAGE:2022-12-01 10:34:10 31733:31733 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2391719Z STAGE:2022-12-01 10:34:10 31732:31732 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2391958Z STAGE:2022-12-01 10:34:10 31731:31731 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2392202Z STAGE:2022-12-01 10:34:10 31733:31733 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2392441Z STAGE:2022-12-01 10:34:10 31732:31732 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2392858Z STAGE:2022-12-01 10:34:10 31731:31731 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 10:34:10 31733:31733 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2392866Z 
2022-12-01T10:43:19.2393113Z STAGE:2022-12-01 10:34:10 31732:31732 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2393350Z STAGE:2022-12-01 10:34:10 31731:31731 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2393748Z STAGE:2022-12-01 10:34:10 31732:31732 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-12-01 10:34:10 31733:31733 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2393753Z 
2022-12-01T10:43:19.2393996Z STAGE:2022-12-01 10:34:10 31732:31732 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2394405Z STAGE:2022-12-01 10:34:10 31731:31731 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 10:34:10 31733:31733 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2394413Z 
2022-12-01T10:43:19.2394468Z ok (1.953s)
2022-12-01T10:43:19.2394487Z 
2022-12-01T10:43:19.2394670Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2394748Z Ran 1 test in 1.953s
2022-12-01T10:43:19.2394753Z 
2022-12-01T10:43:19.2394817Z OK
2022-12-01T10:43:19.2394822Z 
2022-12-01T10:43:19.2394908Z Generating XML reports...
2022-12-01T10:43:19.2395252Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103409.xml
2022-12-01T10:43:19.2395483Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.2395758Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2395884Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2396155Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2396293Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2396298Z 
2022-12-01T10:43:19.2396372Z Running tests...
2022-12-01T10:43:19.2396572Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2396802Z   test_all_reduce_min (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31787
2022-12-01T10:43:19.2396961Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31788
2022-12-01T10:43:19.2397116Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 31789
2022-12-01T10:43:19.2397389Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2397545Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2397811Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2397975Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2398156Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:43:19.2398429Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2398554Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2398836Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2398973Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2399150Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:43:19.2399408Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2399534Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2399812Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2399950Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2400128Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:43:19.2400429Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2400726Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2401017Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2401187Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:43:19.2401339Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:43:19.2401590Z STAGE:2022-12-01 10:34:14 31789:31789 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2401830Z STAGE:2022-12-01 10:34:14 31788:31788 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2401997Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:43:19.2402237Z STAGE:2022-12-01 10:34:14 31787:31787 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2402486Z STAGE:2022-12-01 10:34:14 31787:31787 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2402733Z STAGE:2022-12-01 10:34:14 31789:31789 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2402974Z STAGE:2022-12-01 10:34:14 31788:31788 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2403214Z STAGE:2022-12-01 10:34:14 31787:31787 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2403442Z STAGE:2022-12-01 10:34:14 31789:31789 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2403680Z STAGE:2022-12-01 10:34:14 31788:31788 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2403926Z STAGE:2022-12-01 10:34:14 31787:31787 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2404337Z STAGE:2022-12-01 10:34:14 31788:31788 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 10:34:14 31789:31789 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2404343Z 
2022-12-01T10:43:19.2404613Z STAGE:2022-12-01 10:34:14 31787:31787 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2404853Z STAGE:2022-12-01 10:34:14 31789:31789 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2405116Z STAGE:2022-12-01 10:34:14 31788:31788 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2405524Z STAGE:2022-12-01 10:34:14 31789:31789 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 10:34:14 31788:31788 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2405529Z 
2022-12-01T10:43:19.2405775Z STAGE:2022-12-01 10:34:14 31787:31787 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2405847Z ok (1.956s)
2022-12-01T10:43:19.2405852Z 
2022-12-01T10:43:19.2406034Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2406110Z Ran 1 test in 1.956s
2022-12-01T10:43:19.2406115Z 
2022-12-01T10:43:19.2406179Z OK
2022-12-01T10:43:19.2406183Z 
2022-12-01T10:43:19.2406271Z Generating XML reports...
2022-12-01T10:43:19.2406613Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103412.xml
2022-12-01T10:43:19.2406850Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.2407124Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2407249Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2407533Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2407655Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2407660Z 
2022-12-01T10:43:19.2407734Z Running tests...
2022-12-01T10:43:19.2407933Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2408172Z   test_all_reduce_multigpu (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31843
2022-12-01T10:43:19.2408333Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31844
2022-12-01T10:43:19.2408488Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 31845
2022-12-01T10:43:19.2408763Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2408888Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2409154Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2409290Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2409468Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:43:19.2409746Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2409875Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2410155Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2410291Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2410470Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:43:19.2410742Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2410851Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2411130Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2411264Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2411474Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:43:19.2411803Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2412101Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2412401Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2412573Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:43:19.2412738Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:43:19.2412888Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:43:19.2412986Z skip: CUDA is not available. (1.854s)
2022-12-01T10:43:19.2412994Z 
2022-12-01T10:43:19.2413191Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2413385Z Ran 1 test in 1.854s
2022-12-01T10:43:19.2413393Z 
2022-12-01T10:43:19.2413468Z OK (skipped=1)
2022-12-01T10:43:19.2413473Z 
2022-12-01T10:43:19.2413561Z Generating XML reports...
2022-12-01T10:43:19.2413909Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103416.xml
2022-12-01T10:43:19.2414146Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.2414424Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2414535Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2414819Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2414959Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2414966Z 
2022-12-01T10:43:19.2415043Z Running tests...
2022-12-01T10:43:19.2415240Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2415485Z   test_all_reduce_multigpu_complex (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31896
2022-12-01T10:43:19.2415646Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31897
2022-12-01T10:43:19.2415805Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 31898
2022-12-01T10:43:19.2416064Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2416189Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2416472Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2416609Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2416791Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:43:19.2417062Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2417186Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2417467Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2417604Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2417768Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:43:19.2418038Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2418203Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2418514Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2418649Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2418825Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:43:19.2419125Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2419419Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2419713Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2419866Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:43:19.2420037Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:43:19.2420202Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:43:19.2420300Z skip: CUDA is not available. (1.756s)
2022-12-01T10:43:19.2420305Z 
2022-12-01T10:43:19.2420619Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2420699Z Ran 1 test in 1.756s
2022-12-01T10:43:19.2420704Z 
2022-12-01T10:43:19.2420777Z OK (skipped=1)
2022-12-01T10:43:19.2420781Z 
2022-12-01T10:43:19.2420868Z Generating XML reports...
2022-12-01T10:43:19.2421194Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103419.xml
2022-12-01T10:43:19.2421429Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.2421705Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2421829Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2422111Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2422247Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2422252Z 
2022-12-01T10:43:19.2422328Z Running tests...
2022-12-01T10:43:19.2422525Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2422762Z   test_all_reduce_product (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31949
2022-12-01T10:43:19.2422905Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31950
2022-12-01T10:43:19.2423062Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 31951
2022-12-01T10:43:19.2423334Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2423460Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2423739Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2423877Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2424057Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:43:19.2424327Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2424453Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2424720Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2424912Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2425090Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:43:19.2425397Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2425523Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2425800Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2425937Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2426113Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:43:19.2426396Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2426696Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2426989Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2427157Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:43:19.2427320Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:43:19.2427480Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:43:19.2427889Z STAGE:2022-12-01 10:34:23 31951:31951 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-12-01 10:34:23 31949:31949 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2427896Z 
2022-12-01T10:43:19.2428142Z STAGE:2022-12-01 10:34:23 31950:31950 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2428557Z STAGE:2022-12-01 10:34:23 31951:31951 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 10:34:23 31949:31949 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2428564Z 
2022-12-01T10:43:19.2428817Z STAGE:2022-12-01 10:34:23 31950:31950 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2429218Z STAGE:2022-12-01 10:34:23 31950:31950 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-12-01 10:34:23 31951:31951 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2429224Z 
2022-12-01T10:43:19.2429464Z STAGE:2022-12-01 10:34:23 31949:31949 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2429698Z STAGE:2022-12-01 10:34:23 31950:31950 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2429939Z STAGE:2022-12-01 10:34:23 31949:31949 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2430181Z STAGE:2022-12-01 10:34:23 31951:31951 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2430424Z STAGE:2022-12-01 10:34:23 31950:31950 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2430822Z STAGE:2022-12-01 10:34:23 31951:31951 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-12-01 10:34:23 31949:31949 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2430827Z 
2022-12-01T10:43:19.2431072Z STAGE:2022-12-01 10:34:23 31950:31950 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2431481Z STAGE:2022-12-01 10:34:23 31951:31951 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 10:34:23 31949:31949 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2431487Z 
2022-12-01T10:43:19.2431556Z ok (1.854s)
2022-12-01T10:43:19.2431561Z 
2022-12-01T10:43:19.2431762Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2431881Z Ran 1 test in 1.854s
2022-12-01T10:43:19.2431886Z 
2022-12-01T10:43:19.2431960Z OK
2022-12-01T10:43:19.2431965Z 
2022-12-01T10:43:19.2432051Z Generating XML reports...
2022-12-01T10:43:19.2432393Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103422.xml
2022-12-01T10:43:19.2432626Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.2432899Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2433025Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2433310Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2433447Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2433452Z 
2022-12-01T10:43:19.2433527Z Running tests...
2022-12-01T10:43:19.2433712Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2433952Z   test_all_reduce_result_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32005
2022-12-01T10:43:19.2434109Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32006
2022-12-01T10:43:19.2434265Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 32007
2022-12-01T10:43:19.2434539Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2434664Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2434946Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2435082Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2435248Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:43:19.2435521Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2435645Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2435926Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2436066Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2436243Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:43:19.2436515Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2436640Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2436920Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2437044Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2437222Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:43:19.2437524Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2437824Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2438114Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2438281Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:43:19.2438446Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:43:19.2438640Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:43:19.2438764Z skip: CUDA is not available. (1.855s)
2022-12-01T10:43:19.2438769Z 
2022-12-01T10:43:19.2438953Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2439030Z Ran 1 test in 1.855s
2022-12-01T10:43:19.2439035Z 
2022-12-01T10:43:19.2439107Z OK (skipped=1)
2022-12-01T10:43:19.2439112Z 
2022-12-01T10:43:19.2439199Z Generating XML reports...
2022-12-01T10:43:19.2439543Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103425.xml
2022-12-01T10:43:19.2439775Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.2440047Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2440175Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2440444Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2440583Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2440588Z 
2022-12-01T10:43:19.2440661Z Running tests...
2022-12-01T10:43:19.2440859Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2441090Z   test_all_reduce_sum (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32058
2022-12-01T10:43:19.2441249Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32059
2022-12-01T10:43:19.2441406Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 32060
2022-12-01T10:43:19.2441679Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2441806Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2442073Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2442209Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2442385Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:43:19.2442655Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2442779Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2443061Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2443196Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2443372Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:43:19.2443629Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2443754Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2444032Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2444166Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2444341Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:43:19.2444640Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2444936Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2445255Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2445426Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:43:19.2445599Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:43:19.2445763Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:43:19.2446013Z STAGE:2022-12-01 10:34:30 32058:32058 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2446254Z STAGE:2022-12-01 10:34:30 32059:32059 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2446491Z STAGE:2022-12-01 10:34:30 32060:32060 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2446742Z STAGE:2022-12-01 10:34:30 32058:32058 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2446991Z STAGE:2022-12-01 10:34:30 32060:32060 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2447232Z STAGE:2022-12-01 10:34:30 32059:32059 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2447634Z STAGE:2022-12-01 10:34:30 32060:32060 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-12-01 10:34:30 32058:32058 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2447640Z 
2022-12-01T10:43:19.2447881Z STAGE:2022-12-01 10:34:30 32059:32059 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2448275Z STAGE:2022-12-01 10:34:30 32060:32060 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 10:34:30 32058:32058 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2448294Z 
2022-12-01T10:43:19.2448528Z STAGE:2022-12-01 10:34:30 32059:32059 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2448927Z STAGE:2022-12-01 10:34:30 32060:32060 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-12-01 10:34:30 32058:32058 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2448934Z 
2022-12-01T10:43:19.2449181Z STAGE:2022-12-01 10:34:30 32059:32059 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2449429Z STAGE:2022-12-01 10:34:30 32059:32059 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2449670Z STAGE:2022-12-01 10:34:30 32060:32060 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2449910Z STAGE:2022-12-01 10:34:30 32058:32058 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2449980Z ok (1.856s)
2022-12-01T10:43:19.2449985Z 
2022-12-01T10:43:19.2450180Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2450257Z Ran 1 test in 1.857s
2022-12-01T10:43:19.2450263Z 
2022-12-01T10:43:19.2450310Z OK
2022-12-01T10:43:19.2450314Z 
2022-12-01T10:43:19.2450400Z Generating XML reports...
2022-12-01T10:43:19.2450743Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103429.xml
2022-12-01T10:43:19.2450980Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.2451256Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2451380Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2451663Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2451800Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2451806Z 
2022-12-01T10:43:19.2451879Z Running tests...
2022-12-01T10:43:19.2452061Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2452330Z   test_all_reduce_sum_async (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32114
2022-12-01T10:43:19.2452517Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32115
2022-12-01T10:43:19.2452675Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 32116
2022-12-01T10:43:19.2452947Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2453073Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2453432Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2453575Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2453740Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:43:19.2454020Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2454146Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2454423Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2454562Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2454742Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:43:19.2455012Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2455136Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2455415Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2455535Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2455715Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:43:19.2456017Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2456313Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2456642Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2456810Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:43:19.2456977Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:43:19.2457141Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:43:19.2457394Z STAGE:2022-12-01 10:34:33 32114:32114 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2457624Z STAGE:2022-12-01 10:34:33 32115:32115 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2457870Z STAGE:2022-12-01 10:34:33 32116:32116 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2458116Z STAGE:2022-12-01 10:34:33 32114:32114 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2458359Z STAGE:2022-12-01 10:34:33 32116:32116 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2458599Z STAGE:2022-12-01 10:34:33 32115:32115 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2458996Z STAGE:2022-12-01 10:34:33 32114:32114 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-12-01 10:34:33 32116:32116 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2459002Z 
2022-12-01T10:43:19.2459280Z STAGE:2022-12-01 10:34:33 32115:32115 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2459585Z STAGE:2022-12-01 10:34:33 32114:32114 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2459827Z STAGE:2022-12-01 10:34:33 32116:32116 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2460068Z STAGE:2022-12-01 10:34:33 32115:32115 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2460292Z STAGE:2022-12-01 10:34:33 32116:32116 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2460653Z STAGE:2022-12-01 10:34:33 32114:32114 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2460896Z STAGE:2022-12-01 10:34:33 32115:32115 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2461145Z STAGE:2022-12-01 10:34:33 32114:32114 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2461392Z STAGE:2022-12-01 10:34:33 32115:32115 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2461636Z STAGE:2022-12-01 10:34:33 32116:32116 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2461705Z ok (1.852s)
2022-12-01T10:43:19.2461711Z 
2022-12-01T10:43:19.2461910Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2461972Z Ran 1 test in 1.852s
2022-12-01T10:43:19.2461991Z 
2022-12-01T10:43:19.2462038Z OK
2022-12-01T10:43:19.2462043Z 
2022-12-01T10:43:19.2462128Z Generating XML reports...
2022-12-01T10:43:19.2462467Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103432.xml
2022-12-01T10:43:19.2462702Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.2462979Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2463106Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2463390Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2463527Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2463533Z 
2022-12-01T10:43:19.2463593Z Running tests...
2022-12-01T10:43:19.2463788Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2464029Z   test_all_reduce_sum_complex (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32170
2022-12-01T10:43:19.2464187Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32171
2022-12-01T10:43:19.2464344Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 32172
2022-12-01T10:43:19.2464622Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2464750Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2465032Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2465155Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2465333Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:43:19.2465608Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2465735Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2466014Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2466150Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2466382Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:43:19.2466716Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2466986Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2467097Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2467377Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2467511Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2467686Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:43:19.2467983Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2468284Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2468453Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:43:19.2468621Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:43:19.2468786Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:43:19.2469023Z STAGE:2022-12-01 10:34:37 32170:32170 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2469270Z STAGE:2022-12-01 10:34:37 32172:32172 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2469514Z STAGE:2022-12-01 10:34:37 32171:32171 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2469765Z STAGE:2022-12-01 10:34:37 32170:32170 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2470010Z STAGE:2022-12-01 10:34:37 32172:32172 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2470253Z STAGE:2022-12-01 10:34:37 32171:32171 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2470494Z STAGE:2022-12-01 10:34:37 32170:32170 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2470736Z STAGE:2022-12-01 10:34:37 32171:32171 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2470976Z STAGE:2022-12-01 10:34:37 32172:32172 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2471209Z STAGE:2022-12-01 10:34:37 32172:32172 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2471622Z STAGE:2022-12-01 10:34:37 32170:32170 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 10:34:37 32171:32171 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2471628Z 
2022-12-01T10:43:19.2471868Z STAGE:2022-12-01 10:34:37 32172:32172 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2472107Z STAGE:2022-12-01 10:34:37 32170:32170 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2472343Z STAGE:2022-12-01 10:34:37 32171:32171 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2472588Z STAGE:2022-12-01 10:34:37 32170:32170 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2472832Z STAGE:2022-12-01 10:34:37 32172:32172 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2473078Z STAGE:2022-12-01 10:34:37 32171:32171 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2473147Z ok (1.853s)
2022-12-01T10:43:19.2473152Z 
2022-12-01T10:43:19.2473336Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2473412Z Ran 1 test in 1.853s
2022-12-01T10:43:19.2473417Z 
2022-12-01T10:43:19.2473522Z OK
2022-12-01T10:43:19.2473528Z 
2022-12-01T10:43:19.2473643Z Generating XML reports...
2022-12-01T10:43:19.2473990Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103435.xml
2022-12-01T10:43:19.2474226Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.2474502Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2474628Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2474911Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2475033Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2475039Z 
2022-12-01T10:43:19.2475116Z Running tests...
2022-12-01T10:43:19.2475317Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2475554Z   test_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32226
2022-12-01T10:43:19.2475715Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32227
2022-12-01T10:43:19.2475871Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 32228
2022-12-01T10:43:19.2476149Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2476274Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2476556Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2476678Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2476856Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:43:19.2477132Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2477262Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2477544Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2477682Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2477862Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:43:19.2478137Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2478248Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2478530Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2478669Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2478849Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:43:19.2479157Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2479458Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2479760Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2479928Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:43:19.2480096Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:43:19.2480288Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:43:19.2480387Z skip: CUDA is not available. (1.753s)
2022-12-01T10:43:19.2480422Z 
2022-12-01T10:43:19.2480620Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2480698Z Ran 1 test in 1.753s
2022-12-01T10:43:19.2480703Z 
2022-12-01T10:43:19.2480778Z OK (skipped=1)
2022-12-01T10:43:19.2480783Z 
2022-12-01T10:43:19.2480870Z Generating XML reports...
2022-12-01T10:43:19.2481212Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103439.xml
2022-12-01T10:43:19.2481446Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.2481721Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2481832Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2482119Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2482257Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2482262Z 
2022-12-01T10:43:19.2482336Z Running tests...
2022-12-01T10:43:19.2482531Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2482770Z   test_all_reduce_sum_cuda_async (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32279
2022-12-01T10:43:19.2482926Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32280
2022-12-01T10:43:19.2483084Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 32281
2022-12-01T10:43:19.2483344Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2483470Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2483751Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2483890Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2484068Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:43:19.2484340Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2484463Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2484743Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2484879Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2485044Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:43:19.2485318Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2485441Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2485717Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2485850Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2486027Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:43:19.2486324Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2486623Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2486913Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2487097Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:43:19.2487288Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:43:19.2487450Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:43:19.2487547Z skip: CUDA is not available. (1.852s)
2022-12-01T10:43:19.2487552Z 
2022-12-01T10:43:19.2487749Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2487824Z Ran 1 test in 1.852s
2022-12-01T10:43:19.2487829Z 
2022-12-01T10:43:19.2487903Z OK (skipped=1)
2022-12-01T10:43:19.2487907Z 
2022-12-01T10:43:19.2487992Z Generating XML reports...
2022-12-01T10:43:19.2488319Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103442.xml
2022-12-01T10:43:19.2488552Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.2488832Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2488960Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2489241Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2489378Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2489383Z 
2022-12-01T10:43:19.2489457Z Running tests...
2022-12-01T10:43:19.2489653Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2489896Z   test_all_reduce_sum_cuda_complex (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32332
2022-12-01T10:43:19.2490039Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32333
2022-12-01T10:43:19.2490195Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 32334
2022-12-01T10:43:19.2490467Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2490592Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2490872Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2491008Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2491189Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:43:19.2491460Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2491584Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2491848Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2491986Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2492165Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:43:19.2492435Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2492558Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2492836Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2492972Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2493148Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:43:19.2493510Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2493858Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2494191Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2494358Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:43:19.2494518Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:43:19.2494681Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:43:19.2494779Z skip: CUDA is not available. (1.850s)
2022-12-01T10:43:19.2494785Z 
2022-12-01T10:43:19.2494981Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2495058Z Ran 1 test in 1.850s
2022-12-01T10:43:19.2495063Z 
2022-12-01T10:43:19.2495120Z OK (skipped=1)
2022-12-01T10:43:19.2495140Z 
2022-12-01T10:43:19.2495213Z Generating XML reports...
2022-12-01T10:43:19.2495555Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103445.xml
2022-12-01T10:43:19.2495792Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.2496070Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2496196Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2496475Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2496610Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2496614Z 
2022-12-01T10:43:19.2496688Z Running tests...
2022-12-01T10:43:19.2496868Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2497051Z   test_all_to_all (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.001s)
2022-12-01T10:43:19.2497058Z 
2022-12-01T10:43:19.2497252Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2497328Z Ran 1 test in 0.002s
2022-12-01T10:43:19.2497333Z 
2022-12-01T10:43:19.2497405Z OK (skipped=1)
2022-12-01T10:43:19.2497410Z 
2022-12-01T10:43:19.2497496Z Generating XML reports...
2022-12-01T10:43:19.2497839Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103448.xml
2022-12-01T10:43:19.2498073Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.2498347Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2498458Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2498744Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2498883Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2498888Z 
2022-12-01T10:43:19.2498964Z Running tests...
2022-12-01T10:43:19.2499160Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2499347Z   test_all_to_all_complex (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.001s)
2022-12-01T10:43:19.2499353Z 
2022-12-01T10:43:19.2499545Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2499623Z Ran 1 test in 0.002s
2022-12-01T10:43:19.2499628Z 
2022-12-01T10:43:19.2499685Z OK (skipped=1)
2022-12-01T10:43:19.2499706Z 
2022-12-01T10:43:19.2499776Z Generating XML reports...
2022-12-01T10:43:19.2500117Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103450.xml
2022-12-01T10:43:19.2500533Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.2500847Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2500972Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2501256Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2501391Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2501396Z 
2022-12-01T10:43:19.2501469Z Running tests...
2022-12-01T10:43:19.2501646Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2501835Z   test_all_to_all_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only NCCL supports CUDA all_to_all (0.001s)
2022-12-01T10:43:19.2501841Z 
2022-12-01T10:43:19.2502031Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2502109Z Ran 1 test in 0.002s
2022-12-01T10:43:19.2502114Z 
2022-12-01T10:43:19.2502188Z OK (skipped=1)
2022-12-01T10:43:19.2502193Z 
2022-12-01T10:43:19.2502277Z Generating XML reports...
2022-12-01T10:43:19.2502615Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103451.xml
2022-12-01T10:43:19.2502848Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.2503120Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2503229Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2503512Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2503653Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2503658Z 
2022-12-01T10:43:19.2503732Z Running tests...
2022-12-01T10:43:19.2503927Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2504126Z   test_all_to_all_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip: Only NCCL supports CUDA all_to_all (0.002s)
2022-12-01T10:43:19.2504131Z 
2022-12-01T10:43:19.2504308Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2504384Z Ran 1 test in 0.002s
2022-12-01T10:43:19.2504389Z 
2022-12-01T10:43:19.2504459Z OK (skipped=1)
2022-12-01T10:43:19.2504463Z 
2022-12-01T10:43:19.2504547Z Generating XML reports...
2022-12-01T10:43:19.2504882Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103452.xml
2022-12-01T10:43:19.2505113Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.2505384Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2505510Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2505777Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2505912Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2505916Z 
2022-12-01T10:43:19.2505989Z Running tests...
2022-12-01T10:43:19.2506184Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2506371Z   test_all_to_all_full_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.001s)
2022-12-01T10:43:19.2506376Z 
2022-12-01T10:43:19.2506563Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2506642Z Ran 1 test in 0.002s
2022-12-01T10:43:19.2506646Z 
2022-12-01T10:43:19.2506718Z OK (skipped=1)
2022-12-01T10:43:19.2506722Z 
2022-12-01T10:43:19.2506809Z Generating XML reports...
2022-12-01T10:43:19.2507165Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103454.xml
2022-12-01T10:43:19.2507425Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.2507696Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2507818Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2508100Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2508235Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2508240Z 
2022-12-01T10:43:19.2508311Z Running tests...
2022-12-01T10:43:19.2508506Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2508706Z   test_all_to_all_full_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only NCCL supports CUDA all_to_all (0.001s)
2022-12-01T10:43:19.2508712Z 
2022-12-01T10:43:19.2508890Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2508965Z Ran 1 test in 0.002s
2022-12-01T10:43:19.2508970Z 
2022-12-01T10:43:19.2509040Z OK (skipped=1)
2022-12-01T10:43:19.2509045Z 
2022-12-01T10:43:19.2509127Z Generating XML reports...
2022-12-01T10:43:19.2509465Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103455.xml
2022-12-01T10:43:19.2509694Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.2509965Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2510086Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2510366Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2510490Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2510496Z 
2022-12-01T10:43:19.2510567Z Running tests...
2022-12-01T10:43:19.2510763Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2510944Z   test_all_to_all_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.001s)
2022-12-01T10:43:19.2510949Z 
2022-12-01T10:43:19.2511138Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2511211Z Ran 1 test in 0.002s
2022-12-01T10:43:19.2511215Z 
2022-12-01T10:43:19.2511284Z OK (skipped=1)
2022-12-01T10:43:19.2511288Z 
2022-12-01T10:43:19.2511369Z Generating XML reports...
2022-12-01T10:43:19.2511692Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103456.xml
2022-12-01T10:43:19.2511920Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.2512192Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2512317Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2512595Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2512728Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2512733Z 
2022-12-01T10:43:19.2512806Z Running tests...
2022-12-01T10:43:19.2512996Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2513198Z   test_all_to_all_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-12-01T10:43:19.2513203Z 
2022-12-01T10:43:19.2513380Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2513453Z Ran 1 test in 0.002s
2022-12-01T10:43:19.2513458Z 
2022-12-01T10:43:19.2513559Z OK (skipped=1)
2022-12-01T10:43:19.2513591Z 
2022-12-01T10:43:19.2513673Z Generating XML reports...
2022-12-01T10:43:19.2514009Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103458.xml
2022-12-01T10:43:19.2514240Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.2514513Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2514635Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2514915Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2515037Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2515042Z 
2022-12-01T10:43:19.2515115Z Running tests...
2022-12-01T10:43:19.2515311Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2515520Z   test_all_to_all_single_equal_split (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.001s)
2022-12-01T10:43:19.2515525Z 
2022-12-01T10:43:19.2515716Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2515791Z Ran 1 test in 0.002s
2022-12-01T10:43:19.2515796Z 
2022-12-01T10:43:19.2515866Z OK (skipped=1)
2022-12-01T10:43:19.2515870Z 
2022-12-01T10:43:19.2515952Z Generating XML reports...
2022-12-01T10:43:19.2516286Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103459.xml
2022-12-01T10:43:19.2516505Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.2516776Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2516900Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2517178Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2517314Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2517319Z 
2022-12-01T10:43:19.2517390Z Running tests...
2022-12-01T10:43:19.2517581Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2517796Z   test_all_to_all_single_equal_split_complex (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.001s)
2022-12-01T10:43:19.2517802Z 
2022-12-01T10:43:19.2517994Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2518055Z Ran 1 test in 0.002s
2022-12-01T10:43:19.2518060Z 
2022-12-01T10:43:19.2518129Z OK (skipped=1)
2022-12-01T10:43:19.2518133Z 
2022-12-01T10:43:19.2518215Z Generating XML reports...
2022-12-01T10:43:19.2518552Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103500.xml
2022-12-01T10:43:19.2518783Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.2519056Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2519178Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2519456Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2519578Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2519595Z 
2022-12-01T10:43:19.2519654Z Running tests...
2022-12-01T10:43:19.2519844Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2520084Z   test_all_to_all_single_equal_split_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-12-01T10:43:19.2520090Z 
2022-12-01T10:43:19.2520309Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2520383Z Ran 1 test in 0.002s
2022-12-01T10:43:19.2520388Z 
2022-12-01T10:43:19.2520458Z OK (skipped=1)
2022-12-01T10:43:19.2520462Z 
2022-12-01T10:43:19.2520543Z Generating XML reports...
2022-12-01T10:43:19.2520877Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103502.xml
2022-12-01T10:43:19.2521093Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.2521365Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2521486Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2521766Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2521899Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2521906Z 
2022-12-01T10:43:19.2521977Z Running tests...
2022-12-01T10:43:19.2522172Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2522391Z   test_all_to_all_single_equal_split_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-12-01T10:43:19.2522397Z 
2022-12-01T10:43:19.2522586Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2522646Z Ran 1 test in 0.002s
2022-12-01T10:43:19.2522650Z 
2022-12-01T10:43:19.2522719Z OK (skipped=1)
2022-12-01T10:43:19.2522723Z 
2022-12-01T10:43:19.2522804Z Generating XML reports...
2022-12-01T10:43:19.2523140Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103503.xml
2022-12-01T10:43:19.2523370Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.2523644Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2523766Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2524044Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2524182Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2524187Z 
2022-12-01T10:43:19.2524245Z Running tests...
2022-12-01T10:43:19.2524438Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2524648Z   test_all_to_all_single_equal_split_full_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.001s)
2022-12-01T10:43:19.2524654Z 
2022-12-01T10:43:19.2524840Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2524917Z Ran 1 test in 0.002s
2022-12-01T10:43:19.2524924Z 
2022-12-01T10:43:19.2524993Z OK (skipped=1)
2022-12-01T10:43:19.2524998Z 
2022-12-01T10:43:19.2525079Z Generating XML reports...
2022-12-01T10:43:19.2525415Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103504.xml
2022-12-01T10:43:19.2525636Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.2525905Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2526026Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2526303Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2526437Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2526442Z 
2022-12-01T10:43:19.2526554Z Running tests...
2022-12-01T10:43:19.2526746Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2527000Z   test_all_to_all_single_equal_split_full_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-12-01T10:43:19.2527005Z 
2022-12-01T10:43:19.2527196Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2527257Z Ran 1 test in 0.002s
2022-12-01T10:43:19.2527274Z 
2022-12-01T10:43:19.2527331Z OK (skipped=1)
2022-12-01T10:43:19.2527335Z 
2022-12-01T10:43:19.2527417Z Generating XML reports...
2022-12-01T10:43:19.2527752Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103506.xml
2022-12-01T10:43:19.2527980Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.2528253Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2528374Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2528656Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2528793Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2528798Z 
2022-12-01T10:43:19.2528857Z Running tests...
2022-12-01T10:43:19.2529052Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2529269Z   test_all_to_all_single_equal_split_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.001s)
2022-12-01T10:43:19.2529274Z 
2022-12-01T10:43:19.2529466Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2529542Z Ran 1 test in 0.002s
2022-12-01T10:43:19.2529546Z 
2022-12-01T10:43:19.2529618Z OK (skipped=1)
2022-12-01T10:43:19.2529622Z 
2022-12-01T10:43:19.2529708Z Generating XML reports...
2022-12-01T10:43:19.2530041Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103507.xml
2022-12-01T10:43:19.2530273Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.2530533Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2530654Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2530930Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2531063Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2531068Z 
2022-12-01T10:43:19.2531139Z Running tests...
2022-12-01T10:43:19.2531330Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2531550Z   test_all_to_all_single_equal_split_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-12-01T10:43:19.2531557Z 
2022-12-01T10:43:19.2531745Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2531820Z Ran 1 test in 0.002s
2022-12-01T10:43:19.2531824Z 
2022-12-01T10:43:19.2531881Z OK (skipped=1)
2022-12-01T10:43:19.2531886Z 
2022-12-01T10:43:19.2531968Z Generating XML reports...
2022-12-01T10:43:19.2532303Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103508.xml
2022-12-01T10:43:19.2532532Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.2532803Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2532926Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2533232Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2533489Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2533495Z 
2022-12-01T10:43:19.2533567Z Running tests...
2022-12-01T10:43:19.2533752Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2533962Z   test_all_to_all_single_unequal_split (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.001s)
2022-12-01T10:43:19.2533967Z 
2022-12-01T10:43:19.2534161Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2534236Z Ran 1 test in 0.002s
2022-12-01T10:43:19.2534241Z 
2022-12-01T10:43:19.2534313Z OK (skipped=1)
2022-12-01T10:43:19.2534318Z 
2022-12-01T10:43:19.2534403Z Generating XML reports...
2022-12-01T10:43:19.2534743Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103510.xml
2022-12-01T10:43:19.2534977Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.2535238Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2535362Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2535643Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2535779Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2535783Z 
2022-12-01T10:43:19.2535856Z Running tests...
2022-12-01T10:43:19.2536051Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2536270Z   test_all_to_all_single_unequal_split_complex (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.001s)
2022-12-01T10:43:19.2536275Z 
2022-12-01T10:43:19.2536470Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2536546Z Ran 1 test in 0.002s
2022-12-01T10:43:19.2536550Z 
2022-12-01T10:43:19.2536607Z OK (skipped=1)
2022-12-01T10:43:19.2536611Z 
2022-12-01T10:43:19.2536693Z Generating XML reports...
2022-12-01T10:43:19.2537029Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103511.xml
2022-12-01T10:43:19.2537258Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.2537528Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2537649Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2537927Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2538061Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2538066Z 
2022-12-01T10:43:19.2538139Z Running tests...
2022-12-01T10:43:19.2538318Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2538530Z   test_all_to_all_single_unequal_split_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-12-01T10:43:19.2538535Z 
2022-12-01T10:43:19.2538724Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2538798Z Ran 1 test in 0.002s
2022-12-01T10:43:19.2538802Z 
2022-12-01T10:43:19.2538872Z OK (skipped=1)
2022-12-01T10:43:19.2538876Z 
2022-12-01T10:43:19.2538959Z Generating XML reports...
2022-12-01T10:43:19.2539293Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103512.xml
2022-12-01T10:43:19.2539524Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.2539832Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2539970Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2540249Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2540479Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2540484Z 
2022-12-01T10:43:19.2540558Z Running tests...
2022-12-01T10:43:19.2540755Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2540979Z   test_all_to_all_single_unequal_split_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-12-01T10:43:19.2540984Z 
2022-12-01T10:43:19.2541176Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2541251Z Ran 1 test in 0.002s
2022-12-01T10:43:19.2541255Z 
2022-12-01T10:43:19.2541315Z OK (skipped=1)
2022-12-01T10:43:19.2541333Z 
2022-12-01T10:43:19.2541405Z Generating XML reports...
2022-12-01T10:43:19.2541740Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103514.xml
2022-12-01T10:43:19.2541971Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.2542243Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2542367Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2542648Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2542782Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2542787Z 
2022-12-01T10:43:19.2542859Z Running tests...
2022-12-01T10:43:19.2543040Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2543260Z   test_all_to_all_single_unequal_split_full_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.001s)
2022-12-01T10:43:19.2543267Z 
2022-12-01T10:43:19.2543456Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2543531Z Ran 1 test in 0.002s
2022-12-01T10:43:19.2543536Z 
2022-12-01T10:43:19.2543610Z OK (skipped=1)
2022-12-01T10:43:19.2543614Z 
2022-12-01T10:43:19.2543696Z Generating XML reports...
2022-12-01T10:43:19.2544030Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103515.xml
2022-12-01T10:43:19.2544262Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.2544532Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2544644Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2544923Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2545060Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2545065Z 
2022-12-01T10:43:19.2545136Z Running tests...
2022-12-01T10:43:19.2545327Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2545552Z   test_all_to_all_single_unequal_split_full_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-12-01T10:43:19.2545558Z 
2022-12-01T10:43:19.2545745Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2545818Z Ran 1 test in 0.002s
2022-12-01T10:43:19.2545823Z 
2022-12-01T10:43:19.2545893Z OK (skipped=1)
2022-12-01T10:43:19.2545897Z 
2022-12-01T10:43:19.2545967Z Generating XML reports...
2022-12-01T10:43:19.2546355Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103516.xml
2022-12-01T10:43:19.2546622Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.2546895Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2547018Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2547297Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2547433Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2547438Z 
2022-12-01T10:43:19.2547510Z Running tests...
2022-12-01T10:43:19.2547703Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2547907Z   test_all_to_all_single_unequal_split_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.001s)
2022-12-01T10:43:19.2547913Z 
2022-12-01T10:43:19.2548105Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2548184Z Ran 1 test in 0.002s
2022-12-01T10:43:19.2548190Z 
2022-12-01T10:43:19.2548261Z OK (skipped=1)
2022-12-01T10:43:19.2548265Z 
2022-12-01T10:43:19.2548348Z Generating XML reports...
2022-12-01T10:43:19.2548685Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103518.xml
2022-12-01T10:43:19.2548918Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.2549193Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2549303Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2549586Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2549721Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2549728Z 
2022-12-01T10:43:19.2549800Z Running tests...
2022-12-01T10:43:19.2549994Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2550213Z   test_all_to_all_single_unequal_split_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-12-01T10:43:19.2550219Z 
2022-12-01T10:43:19.2550414Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2550490Z Ran 1 test in 0.002s
2022-12-01T10:43:19.2550494Z 
2022-12-01T10:43:19.2550563Z OK (skipped=1)
2022-12-01T10:43:19.2550568Z 
2022-12-01T10:43:19.2550637Z Generating XML reports...
2022-12-01T10:43:19.2550969Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103519.xml
2022-12-01T10:43:19.2551200Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.2551472Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2551595Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2551872Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2552007Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2552012Z 
2022-12-01T10:43:19.2552084Z Running tests...
2022-12-01T10:43:19.2552276Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2552497Z   test_average_parameters (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32625
2022-12-01T10:43:19.2552654Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32626
2022-12-01T10:43:19.2552836Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 32627
2022-12-01T10:43:19.2553137Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2553256Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2553534Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2553668Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2553847Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:43:19.2554119Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2554229Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2554509Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2554643Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2554821Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:43:19.2555092Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2555216Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2555492Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2555626Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2555787Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:43:19.2556089Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2556386Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2556684Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2556850Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:43:19.2557014Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:43:19.2557176Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:43:19.2557279Z skip: Need at least 2 CUDA devices (1.853s)
2022-12-01T10:43:19.2557284Z 
2022-12-01T10:43:19.2557525Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2557586Z Ran 1 test in 1.854s
2022-12-01T10:43:19.2557591Z 
2022-12-01T10:43:19.2557666Z OK (skipped=1)
2022-12-01T10:43:19.2557671Z 
2022-12-01T10:43:19.2557755Z Generating XML reports...
2022-12-01T10:43:19.2558094Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103520.xml
2022-12-01T10:43:19.2558326Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.2558599Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2558724Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2559004Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2559138Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2559143Z 
2022-12-01T10:43:19.2559203Z Running tests...
2022-12-01T10:43:19.2559431Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2559737Z   test_backend_full_group (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'nccl', 'gloo'} (0.001s)
2022-12-01T10:43:19.2559769Z 
2022-12-01T10:43:19.2559962Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2560037Z Ran 1 test in 0.002s
2022-12-01T10:43:19.2560042Z 
2022-12-01T10:43:19.2560133Z OK (skipped=1)
2022-12-01T10:43:19.2560138Z 
2022-12-01T10:43:19.2560222Z Generating XML reports...
2022-12-01T10:43:19.2560556Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103524.xml
2022-12-01T10:43:19.2560784Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.2561042Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2561166Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2561446Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2561582Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2561587Z 
2022-12-01T10:43:19.2561658Z Running tests...
2022-12-01T10:43:19.2561850Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2562152Z   test_backend_group (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'gloo', 'nccl'} (0.001s)
2022-12-01T10:43:19.2562157Z 
2022-12-01T10:43:19.2562348Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2562409Z Ran 1 test in 0.002s
2022-12-01T10:43:19.2562427Z 
2022-12-01T10:43:19.2562484Z OK (skipped=1)
2022-12-01T10:43:19.2562489Z 
2022-12-01T10:43:19.2562572Z Generating XML reports...
2022-12-01T10:43:19.2562909Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103525.xml
2022-12-01T10:43:19.2563146Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.2563418Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2563543Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2563822Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2563959Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2563964Z 
2022-12-01T10:43:19.2564023Z Running tests...
2022-12-01T10:43:19.2564215Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2564437Z   test_barrier (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32698
2022-12-01T10:43:19.2564596Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32699
2022-12-01T10:43:19.2564753Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 32700
2022-12-01T10:43:19.2565023Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2565144Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2565424Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2565665Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2565832Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:43:19.2566128Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2566421Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2566736Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2566930Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2567137Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:43:19.2567437Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2567598Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2567907Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2568029Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2568236Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:43:19.2568597Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2568928Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2569247Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2569443Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:43:19.2569643Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:43:19.2569868Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:43:19.2569924Z ok (3.155s)
2022-12-01T10:43:19.2569972Z 
2022-12-01T10:43:19.2570157Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2570263Z Ran 1 test in 3.155s
2022-12-01T10:43:19.2570269Z 
2022-12-01T10:43:19.2570381Z OK
2022-12-01T10:43:19.2570387Z 
2022-12-01T10:43:19.2570503Z Generating XML reports...
2022-12-01T10:43:19.2570873Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103526.xml
2022-12-01T10:43:19.2571141Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.2571443Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2571597Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2571866Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2572037Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2572042Z 
2022-12-01T10:43:19.2572145Z Running tests...
2022-12-01T10:43:19.2572400Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2572680Z   test_barrier_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32751
2022-12-01T10:43:19.2572875Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32752
2022-12-01T10:43:19.2573058Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 32753
2022-12-01T10:43:19.2573463Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2573617Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2573889Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2574052Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2574292Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:43:19.2574654Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2574809Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2575119Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2575280Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2575483Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:43:19.2575737Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2575889Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2576195Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2576366Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2576595Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:43:19.2576925Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2577247Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2577602Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2577799Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:43:19.2577947Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:43:19.2578145Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:43:19.2578277Z skip: CUDA is not available. (1.752s)
2022-12-01T10:43:19.2578283Z 
2022-12-01T10:43:19.2578510Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2578633Z Ran 1 test in 1.752s
2022-12-01T10:43:19.2578639Z 
2022-12-01T10:43:19.2578738Z OK (skipped=1)
2022-12-01T10:43:19.2578744Z 
2022-12-01T10:43:19.2578855Z Generating XML reports...
2022-12-01T10:43:19.2579224Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103531.xml
2022-12-01T10:43:19.2579483Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.2579743Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2579904Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2580216Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2580484Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2580491Z 
2022-12-01T10:43:19.2580614Z Running tests...
2022-12-01T10:43:19.2580843Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2581132Z   test_barrier_full_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 336
2022-12-01T10:43:19.2581313Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 337
2022-12-01T10:43:19.2581451Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 338
2022-12-01T10:43:19.2581764Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2581917Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2582281Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2582481Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2582710Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:43:19.2583014Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2583174Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2583490Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2583613Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2583820Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:43:19.2584122Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2584278Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2584587Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2584770Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2585012Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:43:19.2585342Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2585669Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2585952Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2586146Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:43:19.2586340Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:43:19.2586530Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:43:19.2586734Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T10:43:19.2586969Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-12-01T10:43:19.2587169Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T10:43:19.2587494Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:43:19.2587819Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:43:19.2588103Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:43:19.2588202Z ok (3.254s)
2022-12-01T10:43:19.2588208Z 
2022-12-01T10:43:19.2588433Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2588536Z Ran 1 test in 3.254s
2022-12-01T10:43:19.2588542Z 
2022-12-01T10:43:19.2588638Z OK
2022-12-01T10:43:19.2588643Z 
2022-12-01T10:43:19.2588806Z Generating XML reports...
2022-12-01T10:43:19.2589174Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103534.xml
2022-12-01T10:43:19.2589435Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.2589695Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2589878Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2590216Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2590390Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2590395Z 
2022-12-01T10:43:19.2590496Z Running tests...
2022-12-01T10:43:19.2590718Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2591002Z   test_barrier_full_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 398
2022-12-01T10:43:19.2591186Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 399
2022-12-01T10:43:19.2591362Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 400
2022-12-01T10:43:19.2591626Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2591779Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2592097Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2592261Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2592466Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:43:19.2592800Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2592973Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2593280Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2593443Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2593611Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:43:19.2593918Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2594072Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2594379Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2594541Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2594745Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:43:19.2595091Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2595417Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2595753Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2595909Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:43:19.2596099Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:43:19.2596289Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:43:19.2596415Z skip: CUDA is not available. (1.752s)
2022-12-01T10:43:19.2596421Z 
2022-12-01T10:43:19.2596670Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2596774Z Ran 1 test in 1.753s
2022-12-01T10:43:19.2596779Z 
2022-12-01T10:43:19.2596900Z OK (skipped=1)
2022-12-01T10:43:19.2596906Z 
2022-12-01T10:43:19.2597027Z Generating XML reports...
2022-12-01T10:43:19.2597353Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103539.xml
2022-12-01T10:43:19.2597642Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.2597974Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2598128Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2598438Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2598603Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2598608Z 
2022-12-01T10:43:19.2598717Z Running tests...
2022-12-01T10:43:19.2598962Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2599214Z   test_barrier_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 451
2022-12-01T10:43:19.2599358Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 452
2022-12-01T10:43:19.2599534Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 453
2022-12-01T10:43:19.2599836Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2599988Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2600323Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2600494Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2600700Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:43:19.2601019Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2601130Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2601442Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2601610Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2615640Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:43:19.2615980Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2616107Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2616396Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2616534Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2616713Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:43:19.2617023Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2617329Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2617608Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2617775Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:43:19.2617937Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:43:19.2618110Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-12-01T10:43:19.2618282Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T10:43:19.2618443Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:43:19.2618721Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T10:43:19.2619052Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:43:19.2619346Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:43:19.2619621Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:43:19.2619690Z ok (2.753s)
2022-12-01T10:43:19.2619696Z 
2022-12-01T10:43:19.2619898Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2619973Z Ran 1 test in 2.753s
2022-12-01T10:43:19.2619979Z 
2022-12-01T10:43:19.2620039Z OK
2022-12-01T10:43:19.2620043Z 
2022-12-01T10:43:19.2620127Z Generating XML reports...
2022-12-01T10:43:19.2620598Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103542.xml
2022-12-01T10:43:19.2620837Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.2621113Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2621225Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2621508Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2621645Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2621651Z 
2022-12-01T10:43:19.2621723Z Running tests...
2022-12-01T10:43:19.2621917Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2622154Z   test_barrier_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 510
2022-12-01T10:43:19.2622308Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 511
2022-12-01T10:43:19.2622465Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 512
2022-12-01T10:43:19.2622729Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2622851Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2623132Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2623269Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2623445Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:43:19.2623715Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2623840Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2624123Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2624257Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2624420Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:43:19.2624691Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2624813Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2625089Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2625221Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2625451Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:43:19.2625753Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2626089Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2626380Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2626532Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:43:19.2626697Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:43:19.2626858Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:43:19.2626956Z skip: CUDA is not available. (1.851s)
2022-12-01T10:43:19.2626961Z 
2022-12-01T10:43:19.2627163Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2627238Z Ran 1 test in 1.851s
2022-12-01T10:43:19.2627245Z 
2022-12-01T10:43:19.2627316Z OK (skipped=1)
2022-12-01T10:43:19.2627320Z 
2022-12-01T10:43:19.2627404Z Generating XML reports...
2022-12-01T10:43:19.2627746Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103546.xml
2022-12-01T10:43:19.2627964Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.2628239Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2628362Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2628645Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2628781Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2628786Z 
2022-12-01T10:43:19.2628859Z Running tests...
2022-12-01T10:43:19.2629056Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2629299Z   test_barrier_timeout_full_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 563
2022-12-01T10:43:19.2629442Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 564
2022-12-01T10:43:19.2629592Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 565
2022-12-01T10:43:19.2629866Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2629987Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2630267Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2630401Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2630578Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:43:19.2630851Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2630971Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2631237Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2631369Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2631544Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:43:19.2631813Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2631934Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2632240Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2632420Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2632593Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:43:19.2632880Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2633173Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2633464Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2633627Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:43:19.2633788Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:43:19.2633950Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:43:19.2634125Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T10:43:19.2634301Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-12-01T10:43:19.2634471Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T10:43:19.2634765Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:43:19.2635043Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:43:19.2635328Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:43:19.2635396Z ok (2.953s)
2022-12-01T10:43:19.2635403Z 
2022-12-01T10:43:19.2635601Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2635678Z Ran 1 test in 2.954s
2022-12-01T10:43:19.2635684Z 
2022-12-01T10:43:19.2635744Z OK
2022-12-01T10:43:19.2635749Z 
2022-12-01T10:43:19.2635832Z Generating XML reports...
2022-12-01T10:43:19.2636177Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103550.xml
2022-12-01T10:43:19.2636397Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.2636671Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2636796Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2637075Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2637212Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2637219Z 
2022-12-01T10:43:19.2637293Z Running tests...
2022-12-01T10:43:19.2637486Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2637720Z   test_barrier_timeout_global (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 625
2022-12-01T10:43:19.2637873Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 626
2022-12-01T10:43:19.2638012Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 627
2022-12-01T10:43:19.2638280Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2638402Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2638682Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2638844Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2639047Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:43:19.2639318Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2639440Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2639705Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2639841Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2640014Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:43:19.2640282Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2640406Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2640685Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2640817Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2640990Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:43:19.2641289Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2641573Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2641866Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2642028Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:43:19.2642193Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:43:19.2642354Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:43:19.2642531Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:43:19.2642702Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:43:19.2642869Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:43:19.2643161Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2643440Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2643729Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2643795Z ok (3.057s)
2022-12-01T10:43:19.2643802Z 
2022-12-01T10:43:19.2643995Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2644070Z Ran 1 test in 3.057s
2022-12-01T10:43:19.2644075Z 
2022-12-01T10:43:19.2644134Z OK
2022-12-01T10:43:19.2644139Z 
2022-12-01T10:43:19.2644221Z Generating XML reports...
2022-12-01T10:43:19.2644560Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103554.xml
2022-12-01T10:43:19.2644791Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.2645054Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2645176Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2645496Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2645660Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2645666Z 
2022-12-01T10:43:19.2645737Z Running tests...
2022-12-01T10:43:19.2645930Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2646166Z   test_barrier_timeout_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 688
2022-12-01T10:43:19.2646318Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 690
2022-12-01T10:43:19.2646468Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 691
2022-12-01T10:43:19.2646727Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2646850Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2647130Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2647266Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2647441Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:43:19.2647711Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2647832Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2648109Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2648231Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2648407Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:43:19.2648676Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2648801Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2649078Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2649213Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2649385Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:43:19.2649681Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2649974Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2650248Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2650416Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:43:19.2650580Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:43:19.2650741Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:43:19.2650914Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T10:43:19.2651087Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-12-01T10:43:19.2651256Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T10:43:19.2651546Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:43:19.2651837Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:43:19.2652140Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:43:19.2652235Z ok (6.859s)
2022-12-01T10:43:19.2652241Z 
2022-12-01T10:43:19.2652436Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2652510Z Ran 1 test in 6.859s
2022-12-01T10:43:19.2652516Z 
2022-12-01T10:43:19.2652574Z OK
2022-12-01T10:43:19.2652579Z 
2022-12-01T10:43:19.2652662Z Generating XML reports...
2022-12-01T10:43:19.2653004Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103559.xml
2022-12-01T10:43:19.2653235Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.2653593Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2653753Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2654034Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2654173Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2654178Z 
2022-12-01T10:43:19.2654251Z Running tests...
2022-12-01T10:43:19.2654443Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2654678Z   test_batch_isend_irecv_gloo (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 748
2022-12-01T10:43:19.2654832Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 749
2022-12-01T10:43:19.2654988Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 750
2022-12-01T10:43:19.2655249Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2655374Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2655652Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2655790Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2655969Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:43:19.2656240Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2656362Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2656640Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2656776Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2656939Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:43:19.2657238Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2657506Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2657627Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2657904Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2658037Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2658209Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:43:19.2658506Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2658835Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2659015Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:43:19.2659177Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:43:19.2659337Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:43:19.2659403Z ok (1.951s)
2022-12-01T10:43:19.2659408Z 
2022-12-01T10:43:19.2659603Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2659677Z Ran 1 test in 1.951s
2022-12-01T10:43:19.2659682Z 
2022-12-01T10:43:19.2659741Z OK
2022-12-01T10:43:19.2659746Z 
2022-12-01T10:43:19.2659827Z Generating XML reports...
2022-12-01T10:43:19.2660153Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103607.xml
2022-12-01T10:43:19.2660483Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.2660762Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2660889Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2661170Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2661305Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2661311Z 
2022-12-01T10:43:19.2661383Z Running tests...
2022-12-01T10:43:19.2661580Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2661817Z   test_batch_isend_irecv_gloo_tags (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 801
2022-12-01T10:43:19.2661958Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 802
2022-12-01T10:43:19.2662112Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 803
2022-12-01T10:43:19.2662385Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2662509Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2662787Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2662921Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2663099Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:43:19.2663396Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2663663Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2663774Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2664049Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2664184Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2664358Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:43:19.2664628Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2664751Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2665028Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2665161Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2665322Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:43:19.2665668Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2666004Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2666168Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:43:19.2666331Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:43:19.2666486Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:43:19.2666554Z ok (1.957s)
2022-12-01T10:43:19.2666559Z 
2022-12-01T10:43:19.2666754Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2666828Z Ran 1 test in 1.958s
2022-12-01T10:43:19.2666833Z 
2022-12-01T10:43:19.2666880Z OK
2022-12-01T10:43:19.2666884Z 
2022-12-01T10:43:19.2666969Z Generating XML reports...
2022-12-01T10:43:19.2667311Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103610.xml
2022-12-01T10:43:19.2667545Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.2667817Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2667942Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2668221Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2668355Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2668361Z 
2022-12-01T10:43:19.2668432Z Running tests...
2022-12-01T10:43:19.2668613Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2668814Z   test_batch_isend_irecv_mixed_backend_err (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s)
2022-12-01T10:43:19.2668821Z 
2022-12-01T10:43:19.2669013Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2669088Z Ran 1 test in 0.002s
2022-12-01T10:43:19.2669093Z 
2022-12-01T10:43:19.2669163Z OK (skipped=1)
2022-12-01T10:43:19.2669168Z 
2022-12-01T10:43:19.2669249Z Generating XML reports...
2022-12-01T10:43:19.2669589Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103614.xml
2022-12-01T10:43:19.2669823Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.2670082Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2670204Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2670486Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2670621Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2670626Z 
2022-12-01T10:43:19.2670696Z Running tests...
2022-12-01T10:43:19.2670888Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2671073Z   test_batch_isend_irecv_nccl (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s)
2022-12-01T10:43:19.2671078Z 
2022-12-01T10:43:19.2671270Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2671343Z Ran 1 test in 0.003s
2022-12-01T10:43:19.2671347Z 
2022-12-01T10:43:19.2671405Z OK (skipped=1)
2022-12-01T10:43:19.2671409Z 
2022-12-01T10:43:19.2671491Z Generating XML reports...
2022-12-01T10:43:19.2671829Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103615.xml
2022-12-01T10:43:19.2672087Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.2672385Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2672506Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2672786Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2672919Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2672924Z 
2022-12-01T10:43:19.2672994Z Running tests...
2022-12-01T10:43:19.2673173Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2673371Z   test_batch_isend_irecv_no_rank_zero_nccl (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s)
2022-12-01T10:43:19.2673376Z 
2022-12-01T10:43:19.2673564Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2673639Z Ran 1 test in 0.002s
2022-12-01T10:43:19.2673647Z 
2022-12-01T10:43:19.2673716Z OK (skipped=1)
2022-12-01T10:43:19.2673721Z 
2022-12-01T10:43:19.2673802Z Generating XML reports...
2022-12-01T10:43:19.2674143Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103616.xml
2022-12-01T10:43:19.2674377Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.2674635Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2674757Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2675033Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2675165Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2675169Z 
2022-12-01T10:43:19.2675242Z Running tests...
2022-12-01T10:43:19.2675434Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2675623Z   test_batch_isend_irecv_op_err (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s)
2022-12-01T10:43:19.2675628Z 
2022-12-01T10:43:19.2675818Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2675891Z Ran 1 test in 0.002s
2022-12-01T10:43:19.2675897Z 
2022-12-01T10:43:19.2675954Z OK (skipped=1)
2022-12-01T10:43:19.2675959Z 
2022-12-01T10:43:19.2676040Z Generating XML reports...
2022-12-01T10:43:19.2676377Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103618.xml
2022-12-01T10:43:19.2676607Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.2676878Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2677003Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2677281Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2677414Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2677419Z 
2022-12-01T10:43:19.2677490Z Running tests...
2022-12-01T10:43:19.2677673Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2677863Z   test_batch_isend_irecv_op_list_err (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s)
2022-12-01T10:43:19.2677868Z 
2022-12-01T10:43:19.2678056Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2678130Z Ran 1 test in 0.002s
2022-12-01T10:43:19.2678135Z 
2022-12-01T10:43:19.2678205Z OK (skipped=1)
2022-12-01T10:43:19.2678209Z 
2022-12-01T10:43:19.2678290Z Generating XML reports...
2022-12-01T10:43:19.2678660Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103619.xml
2022-12-01T10:43:19.2678961Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.2679234Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2679344Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2679624Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2679760Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2679765Z 
2022-12-01T10:43:19.2679836Z Running tests...
2022-12-01T10:43:19.2680030Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2680232Z   test_batch_isend_irecv_ring_exchange_nccl (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s)
2022-12-01T10:43:19.2680239Z 
2022-12-01T10:43:19.2680433Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2680507Z Ran 1 test in 0.002s
2022-12-01T10:43:19.2680512Z 
2022-12-01T10:43:19.2680570Z OK (skipped=1)
2022-12-01T10:43:19.2680587Z 
2022-12-01T10:43:19.2680656Z Generating XML reports...
2022-12-01T10:43:19.2680995Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103620.xml
2022-12-01T10:43:19.2681227Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.2681498Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2681623Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2681904Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2682037Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2682044Z 
2022-12-01T10:43:19.2682114Z Running tests...
2022-12-01T10:43:19.2682295Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2682486Z   test_batch_isend_irecv_self_nccl (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s)
2022-12-01T10:43:19.2682491Z 
2022-12-01T10:43:19.2682679Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2682751Z Ran 1 test in 0.002s
2022-12-01T10:43:19.2682756Z 
2022-12-01T10:43:19.2682825Z OK (skipped=1)
2022-12-01T10:43:19.2682829Z 
2022-12-01T10:43:19.2682911Z Generating XML reports...
2022-12-01T10:43:19.2683244Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103622.xml
2022-12-01T10:43:19.2683475Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.2683750Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2683863Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2684142Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2684274Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2684279Z 
2022-12-01T10:43:19.2684351Z Running tests...
2022-12-01T10:43:19.2684544Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2684733Z   test_batch_isend_irecv_tensor_err (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s)
2022-12-01T10:43:19.2684739Z 
2022-12-01T10:43:19.2684927Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2684999Z Ran 1 test in 0.002s
2022-12-01T10:43:19.2685036Z 
2022-12-01T10:43:19.2685095Z OK (skipped=1)
2022-12-01T10:43:19.2685139Z 
2022-12-01T10:43:19.2685209Z Generating XML reports...
2022-12-01T10:43:19.2685547Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103623.xml
2022-12-01T10:43:19.2685779Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.2686052Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2686173Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2686454Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2686587Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2686591Z 
2022-12-01T10:43:19.2686662Z Running tests...
2022-12-01T10:43:19.2686847Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2687072Z   test_broadcast (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 934
2022-12-01T10:43:19.2687223Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 935
2022-12-01T10:43:19.2687372Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 936
2022-12-01T10:43:19.2687642Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2687763Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2688042Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2688177Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2688356Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:43:19.2688615Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2688737Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2689013Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2689148Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2689324Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:43:19.2689622Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2689889Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2690013Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2690276Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2690410Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2690583Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:43:19.2690877Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2691171Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2691335Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:43:19.2691497Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:43:19.2691687Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:43:19.2691954Z STAGE:2022-12-01 10:36:26 934:934 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2692183Z STAGE:2022-12-01 10:36:26 936:936 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2692421Z STAGE:2022-12-01 10:36:26 935:935 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2692661Z STAGE:2022-12-01 10:36:26 936:936 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2692902Z STAGE:2022-12-01 10:36:26 934:934 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2693142Z STAGE:2022-12-01 10:36:26 935:935 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2693453Z STAGE:2022-12-01 10:36:26 934:934 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2693698Z STAGE:2022-12-01 10:36:26 936:936 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2693932Z STAGE:2022-12-01 10:36:26 935:935 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2694340Z STAGE:2022-12-01 10:36:26 934:934 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 10:36:26 935:935 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2694346Z 
2022-12-01T10:43:19.2694586Z STAGE:2022-12-01 10:36:26 936:936 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2694808Z STAGE:2022-12-01 10:36:26 935:935 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2695041Z STAGE:2022-12-01 10:36:26 936:936 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2695282Z STAGE:2022-12-01 10:36:26 935:935 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2695515Z STAGE:2022-12-01 10:36:26 934:934 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2695921Z STAGE:2022-12-01 10:36:26 936:936 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 10:36:26 934:934 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2695928Z 
2022-12-01T10:43:19.2696161Z STAGE:2022-12-01 10:36:26 935:935 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2696398Z STAGE:2022-12-01 10:36:26 936:936 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2696630Z STAGE:2022-12-01 10:36:26 934:934 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2696870Z STAGE:2022-12-01 10:36:26 934:934 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2697097Z STAGE:2022-12-01 10:36:26 935:935 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2697334Z STAGE:2022-12-01 10:36:26 936:936 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2697570Z STAGE:2022-12-01 10:36:26 934:934 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2697802Z STAGE:2022-12-01 10:36:26 936:936 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2698031Z STAGE:2022-12-01 10:36:26 935:935 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2698271Z STAGE:2022-12-01 10:36:26 936:936 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2698508Z STAGE:2022-12-01 10:36:26 935:935 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2698743Z STAGE:2022-12-01 10:36:26 934:934 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2698973Z STAGE:2022-12-01 10:36:26 935:935 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2699193Z STAGE:2022-12-01 10:36:26 936:936 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2699465Z STAGE:2022-12-01 10:36:26 934:934 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2699729Z STAGE:2022-12-01 10:36:26 935:935 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2699966Z STAGE:2022-12-01 10:36:26 936:936 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2700201Z STAGE:2022-12-01 10:36:26 934:934 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2700557Z STAGE:2022-12-01 10:36:26 936:936 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2700797Z STAGE:2022-12-01 10:36:26 935:935 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2701032Z STAGE:2022-12-01 10:36:26 934:934 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2701437Z STAGE:2022-12-01 10:36:26 934:934 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 10:36:26 935:935 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2701443Z 
2022-12-01T10:43:19.2701670Z STAGE:2022-12-01 10:36:26 936:936 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2701901Z STAGE:2022-12-01 10:36:26 934:934 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2702132Z STAGE:2022-12-01 10:36:26 936:936 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2702366Z STAGE:2022-12-01 10:36:26 935:935 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2702766Z STAGE:2022-12-01 10:36:26 935:935 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 10:36:26 936:936 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2702773Z 
2022-12-01T10:43:19.2703011Z STAGE:2022-12-01 10:36:26 934:934 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2703247Z STAGE:2022-12-01 10:36:26 935:935 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2703482Z STAGE:2022-12-01 10:36:26 936:936 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2703715Z STAGE:2022-12-01 10:36:26 934:934 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2703953Z STAGE:2022-12-01 10:36:26 935:935 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2704182Z STAGE:2022-12-01 10:36:26 936:936 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2704417Z STAGE:2022-12-01 10:36:26 934:934 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2704650Z STAGE:2022-12-01 10:36:26 936:936 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2704881Z STAGE:2022-12-01 10:36:26 935:935 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2705116Z STAGE:2022-12-01 10:36:26 934:934 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2705351Z STAGE:2022-12-01 10:36:26 934:934 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2705591Z STAGE:2022-12-01 10:36:26 935:935 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2705829Z STAGE:2022-12-01 10:36:26 936:936 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2706063Z STAGE:2022-12-01 10:36:26 934:934 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2706282Z STAGE:2022-12-01 10:36:26 935:935 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2706511Z STAGE:2022-12-01 10:36:26 936:936 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2706747Z STAGE:2022-12-01 10:36:26 934:934 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2707031Z STAGE:2022-12-01 10:36:26 935:935 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2707269Z STAGE:2022-12-01 10:36:26 936:936 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2707539Z STAGE:2022-12-01 10:36:26 935:935 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2707770Z STAGE:2022-12-01 10:36:26 934:934 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2708003Z STAGE:2022-12-01 10:36:26 936:936 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2708228Z STAGE:2022-12-01 10:36:26 936:936 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2708464Z STAGE:2022-12-01 10:36:26 934:934 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2708701Z STAGE:2022-12-01 10:36:26 935:935 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2708932Z STAGE:2022-12-01 10:36:26 936:936 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2709167Z STAGE:2022-12-01 10:36:26 934:934 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2709401Z STAGE:2022-12-01 10:36:26 935:935 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2709801Z STAGE:2022-12-01 10:36:26 934:934 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 10:36:26 936:936 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2709806Z 
2022-12-01T10:43:19.2710047Z STAGE:2022-12-01 10:36:26 935:935 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2710279Z STAGE:2022-12-01 10:36:26 934:934 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2710509Z STAGE:2022-12-01 10:36:26 936:936 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2710727Z STAGE:2022-12-01 10:36:26 935:935 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2710967Z STAGE:2022-12-01 10:36:26 935:935 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2711208Z STAGE:2022-12-01 10:36:26 934:934 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2711446Z STAGE:2022-12-01 10:36:26 936:936 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2711679Z STAGE:2022-12-01 10:36:26 935:935 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2711910Z STAGE:2022-12-01 10:36:26 934:934 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2712142Z STAGE:2022-12-01 10:36:26 936:936 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2712377Z STAGE:2022-12-01 10:36:26 936:936 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2712614Z STAGE:2022-12-01 10:36:26 935:935 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2712841Z STAGE:2022-12-01 10:36:26 934:934 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2713074Z STAGE:2022-12-01 10:36:26 936:936 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2713304Z STAGE:2022-12-01 10:36:26 935:935 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2713534Z STAGE:2022-12-01 10:36:26 934:934 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2713932Z STAGE:2022-12-01 10:36:26 934:934 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 10:36:26 935:935 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2713937Z 
2022-12-01T10:43:19.2714173Z STAGE:2022-12-01 10:36:26 936:936 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2714404Z STAGE:2022-12-01 10:36:26 934:934 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2714670Z STAGE:2022-12-01 10:36:26 935:935 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2714938Z STAGE:2022-12-01 10:36:26 936:936 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2715163Z STAGE:2022-12-01 10:36:26 934:934 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2715399Z STAGE:2022-12-01 10:36:26 935:935 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2715635Z STAGE:2022-12-01 10:36:26 936:936 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2715865Z STAGE:2022-12-01 10:36:26 935:935 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2716097Z STAGE:2022-12-01 10:36:26 934:934 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2716328Z STAGE:2022-12-01 10:36:26 936:936 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2716567Z STAGE:2022-12-01 10:36:26 936:936 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2716807Z STAGE:2022-12-01 10:36:26 934:934 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2717044Z STAGE:2022-12-01 10:36:26 935:935 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2717099Z ok (1.854s)
2022-12-01T10:43:19.2717104Z 
2022-12-01T10:43:19.2717301Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2717376Z Ran 1 test in 1.855s
2022-12-01T10:43:19.2717381Z 
2022-12-01T10:43:19.2717440Z OK
2022-12-01T10:43:19.2717445Z 
2022-12-01T10:43:19.2717528Z Generating XML reports...
2022-12-01T10:43:19.2717873Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103624.xml
2022-12-01T10:43:19.2718105Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.2718381Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2718495Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2718775Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2718909Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2718914Z 
2022-12-01T10:43:19.2718984Z Running tests...
2022-12-01T10:43:19.2719178Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2719861Z   test_broadcast_cuda (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/81028 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.234s)
2022-12-01T10:43:19.2719867Z 
2022-12-01T10:43:19.2720062Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2720138Z Ran 1 test in 0.234s
2022-12-01T10:43:19.2720142Z 
2022-12-01T10:43:19.2720212Z OK (skipped=1)
2022-12-01T10:43:19.2720216Z 
2022-12-01T10:43:19.2720299Z Generating XML reports...
2022-12-01T10:43:19.2720624Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103628.xml
2022-12-01T10:43:19.2720856Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.2721127Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2721249Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2721528Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2721695Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2721700Z 
2022-12-01T10:43:19.2721799Z Running tests...
2022-12-01T10:43:19.2721994Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2722226Z   test_broadcast_full_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1000
2022-12-01T10:43:19.2722369Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1001
2022-12-01T10:43:19.2722523Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1002
2022-12-01T10:43:19.2722792Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2722914Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2723193Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2723329Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2723507Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:43:19.2723778Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2723887Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2724165Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2724297Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2724472Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:43:19.2724740Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2724863Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2725142Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2725278Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2725454Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:43:19.2725739Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2726035Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2726326Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2726491Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:43:19.2726656Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:43:19.2726819Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:43:19.2726994Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T10:43:19.2727166Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-12-01T10:43:19.2727335Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T10:43:19.2727614Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:43:19.2727908Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:43:19.2728230Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:43:19.2728633Z STAGE:2022-12-01 10:36:31 1000:1000 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-12-01 10:36:31 1002:1002 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2728678Z 
2022-12-01T10:43:19.2728931Z STAGE:2022-12-01 10:36:31 1002:1002 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2729170Z STAGE:2022-12-01 10:36:31 1002:1002 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2729410Z STAGE:2022-12-01 10:36:31 1001:1001 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2729653Z STAGE:2022-12-01 10:36:31 1001:1001 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2729894Z STAGE:2022-12-01 10:36:31 1000:1000 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2730131Z STAGE:2022-12-01 10:36:31 1000:1000 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2730357Z STAGE:2022-12-01 10:36:31 1001:1001 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2730601Z STAGE:2022-12-01 10:36:31 1001:1001 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2730845Z STAGE:2022-12-01 10:36:31 1002:1002 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2731088Z STAGE:2022-12-01 10:36:31 1000:1000 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2731325Z STAGE:2022-12-01 10:36:31 1001:1001 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2731560Z STAGE:2022-12-01 10:36:31 1002:1002 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2731798Z STAGE:2022-12-01 10:36:31 1000:1000 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2732036Z STAGE:2022-12-01 10:36:31 1001:1001 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2732443Z STAGE:2022-12-01 10:36:31 1000:1000 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 10:36:31 1002:1002 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2732450Z 
2022-12-01T10:43:19.2732673Z STAGE:2022-12-01 10:36:31 1001:1001 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2732910Z STAGE:2022-12-01 10:36:31 1002:1002 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2733142Z STAGE:2022-12-01 10:36:31 1000:1000 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2733627Z STAGE:2022-12-01 10:36:31 1001:1001 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 10:36:31 1000:1000 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2733633Z 
2022-12-01T10:43:19.2733879Z STAGE:2022-12-01 10:36:31 1002:1002 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2734118Z STAGE:2022-12-01 10:36:31 1000:1000 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2734355Z STAGE:2022-12-01 10:36:31 1001:1001 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2734595Z STAGE:2022-12-01 10:36:31 1002:1002 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2734837Z STAGE:2022-12-01 10:36:31 1000:1000 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2735076Z STAGE:2022-12-01 10:36:31 1001:1001 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2735299Z STAGE:2022-12-01 10:36:31 1002:1002 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2735537Z STAGE:2022-12-01 10:36:31 1001:1001 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2735769Z STAGE:2022-12-01 10:36:31 1000:1000 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2736041Z STAGE:2022-12-01 10:36:31 1002:1002 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2736311Z STAGE:2022-12-01 10:36:31 1002:1002 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2736546Z STAGE:2022-12-01 10:36:31 1000:1000 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2736788Z STAGE:2022-12-01 10:36:31 1001:1001 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2737023Z STAGE:2022-12-01 10:36:31 1002:1002 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2737258Z STAGE:2022-12-01 10:36:31 1000:1000 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2737479Z STAGE:2022-12-01 10:36:31 1001:1001 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2737717Z STAGE:2022-12-01 10:36:31 1002:1002 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2738121Z STAGE:2022-12-01 10:36:31 1000:1000 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 10:36:31 1001:1001 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2738129Z 
2022-12-01T10:43:19.2738368Z STAGE:2022-12-01 10:36:31 1000:1000 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2738603Z STAGE:2022-12-01 10:36:31 1002:1002 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2738837Z STAGE:2022-12-01 10:36:31 1001:1001 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2739077Z STAGE:2022-12-01 10:36:31 1001:1001 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2739314Z STAGE:2022-12-01 10:36:31 1002:1002 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2739550Z STAGE:2022-12-01 10:36:31 1000:1000 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2739775Z STAGE:2022-12-01 10:36:31 1001:1001 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2740014Z STAGE:2022-12-01 10:36:31 1002:1002 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2740250Z STAGE:2022-12-01 10:36:31 1000:1000 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2740595Z STAGE:2022-12-01 10:36:31 1001:1001 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2741001Z STAGE:2022-12-01 10:36:31 1002:1002 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 10:36:31 1000:1000 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2741007Z 
2022-12-01T10:43:19.2741242Z STAGE:2022-12-01 10:36:31 1002:1002 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2741475Z STAGE:2022-12-01 10:36:31 1001:1001 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2741712Z STAGE:2022-12-01 10:36:31 1000:1000 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2741962Z STAGE:2022-12-01 10:36:31 1001:1001 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2742198Z STAGE:2022-12-01 10:36:31 1000:1000 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2742420Z STAGE:2022-12-01 10:36:31 1002:1002 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2742658Z STAGE:2022-12-01 10:36:31 1000:1000 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2742894Z STAGE:2022-12-01 10:36:31 1001:1001 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2743131Z STAGE:2022-12-01 10:36:31 1002:1002 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2743368Z STAGE:2022-12-01 10:36:31 1000:1000 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2743650Z STAGE:2022-12-01 10:36:31 1001:1001 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2743926Z STAGE:2022-12-01 10:36:31 1002:1002 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2744162Z STAGE:2022-12-01 10:36:31 1001:1001 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2744383Z STAGE:2022-12-01 10:36:31 1000:1000 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2744617Z STAGE:2022-12-01 10:36:31 1002:1002 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2744852Z STAGE:2022-12-01 10:36:31 1002:1002 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2745090Z STAGE:2022-12-01 10:36:31 1000:1000 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2745331Z STAGE:2022-12-01 10:36:31 1001:1001 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2745568Z STAGE:2022-12-01 10:36:31 1002:1002 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2745805Z STAGE:2022-12-01 10:36:31 1000:1000 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2746042Z STAGE:2022-12-01 10:36:31 1001:1001 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2746283Z STAGE:2022-12-01 10:36:31 1002:1002 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2746506Z STAGE:2022-12-01 10:36:31 1001:1001 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2746741Z STAGE:2022-12-01 10:36:31 1000:1000 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2746975Z STAGE:2022-12-01 10:36:31 1000:1000 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2747208Z STAGE:2022-12-01 10:36:31 1002:1002 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2747450Z STAGE:2022-12-01 10:36:31 1001:1001 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2747688Z STAGE:2022-12-01 10:36:31 1001:1001 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2748092Z STAGE:2022-12-01 10:36:31 1000:1000 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 10:36:31 1002:1002 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2748098Z 
2022-12-01T10:43:19.2748333Z STAGE:2022-12-01 10:36:31 1001:1001 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2748720Z STAGE:2022-12-01 10:36:31 1002:1002 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-12-01 10:36:31 1000:1000 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2748725Z 
2022-12-01T10:43:19.2748966Z STAGE:2022-12-01 10:36:31 1001:1001 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2749196Z STAGE:2022-12-01 10:36:31 1002:1002 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2749434Z STAGE:2022-12-01 10:36:31 1000:1000 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2749825Z STAGE:2022-12-01 10:36:31 1001:1001 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-12-01 10:36:31 1002:1002 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2749830Z 
2022-12-01T10:43:19.2750066Z STAGE:2022-12-01 10:36:31 1000:1000 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2750307Z STAGE:2022-12-01 10:36:31 1001:1001 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2750545Z STAGE:2022-12-01 10:36:31 1000:1000 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2750786Z STAGE:2022-12-01 10:36:31 1002:1002 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2751053Z STAGE:2022-12-01 10:36:31 1000:1000 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2751315Z STAGE:2022-12-01 10:36:31 1001:1001 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2751549Z STAGE:2022-12-01 10:36:31 1002:1002 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2751773Z STAGE:2022-12-01 10:36:31 1000:1000 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2752014Z STAGE:2022-12-01 10:36:31 1001:1001 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2752250Z STAGE:2022-12-01 10:36:31 1002:1002 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2752484Z STAGE:2022-12-01 10:36:31 1001:1001 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2752718Z STAGE:2022-12-01 10:36:31 1000:1000 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2752955Z STAGE:2022-12-01 10:36:31 1002:1002 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2753194Z STAGE:2022-12-01 10:36:31 1002:1002 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2753428Z STAGE:2022-12-01 10:36:31 1000:1000 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2753662Z STAGE:2022-12-01 10:36:31 1001:1001 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2753717Z ok (2.056s)
2022-12-01T10:43:19.2753722Z 
2022-12-01T10:43:19.2753917Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2753991Z Ran 1 test in 2.056s
2022-12-01T10:43:19.2753995Z 
2022-12-01T10:43:19.2754054Z OK
2022-12-01T10:43:19.2754059Z 
2022-12-01T10:43:19.2754142Z Generating XML reports...
2022-12-01T10:43:19.2754484Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103629.xml
2022-12-01T10:43:19.2754717Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.2754995Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2755105Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2755386Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2755521Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2755526Z 
2022-12-01T10:43:19.2755597Z Running tests...
2022-12-01T10:43:19.2755791Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2756020Z   test_broadcast_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1065
2022-12-01T10:43:19.2756178Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1066
2022-12-01T10:43:19.2756330Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1067
2022-12-01T10:43:19.2756605Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2756715Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2756995Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2757128Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2757305Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:43:19.2757575Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2757697Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2758005Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2758166Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2758330Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:43:19.2758599Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2758720Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2758996Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2759129Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2759302Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:43:19.2759604Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2759905Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2760194Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2760346Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:43:19.2760504Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:43:19.2760679Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T10:43:19.2760837Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:43:19.2761007Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-12-01T10:43:19.2761183Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T10:43:19.2761476Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:43:19.2761768Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:43:19.2762011Z STAGE:2022-12-01 10:36:34 1066:1066 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2762292Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:43:19.2762530Z STAGE:2022-12-01 10:36:34 1067:1067 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2762775Z STAGE:2022-12-01 10:36:34 1067:1067 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2763016Z STAGE:2022-12-01 10:36:34 1066:1066 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2763256Z STAGE:2022-12-01 10:36:34 1066:1066 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2763497Z STAGE:2022-12-01 10:36:34 1067:1067 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2763741Z STAGE:2022-12-01 10:36:34 1067:1067 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2763980Z STAGE:2022-12-01 10:36:34 1066:1066 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2764217Z STAGE:2022-12-01 10:36:34 1067:1067 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2764437Z STAGE:2022-12-01 10:36:34 1066:1066 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2764680Z STAGE:2022-12-01 10:36:34 1066:1066 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2764948Z STAGE:2022-12-01 10:36:34 1067:1067 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2765186Z STAGE:2022-12-01 10:36:34 1066:1066 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2765449Z STAGE:2022-12-01 10:36:34 1067:1067 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2765690Z STAGE:2022-12-01 10:36:34 1067:1067 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2765928Z STAGE:2022-12-01 10:36:34 1066:1066 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2766165Z STAGE:2022-12-01 10:36:34 1067:1067 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2766398Z STAGE:2022-12-01 10:36:34 1066:1066 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2766627Z STAGE:2022-12-01 10:36:34 1066:1066 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2766867Z STAGE:2022-12-01 10:36:34 1067:1067 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2767101Z STAGE:2022-12-01 10:36:34 1066:1066 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2767338Z STAGE:2022-12-01 10:36:34 1067:1067 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2767578Z STAGE:2022-12-01 10:36:34 1067:1067 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2767815Z STAGE:2022-12-01 10:36:34 1066:1066 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2768050Z STAGE:2022-12-01 10:36:34 1067:1067 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2768281Z STAGE:2022-12-01 10:36:34 1066:1066 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2768512Z STAGE:2022-12-01 10:36:34 1066:1066 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2768752Z STAGE:2022-12-01 10:36:34 1067:1067 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2768988Z STAGE:2022-12-01 10:36:34 1066:1066 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2769224Z STAGE:2022-12-01 10:36:34 1067:1067 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2769465Z STAGE:2022-12-01 10:36:34 1067:1067 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2769703Z STAGE:2022-12-01 10:36:34 1066:1066 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2769939Z STAGE:2022-12-01 10:36:34 1067:1067 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2770171Z STAGE:2022-12-01 10:36:34 1066:1066 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2770412Z STAGE:2022-12-01 10:36:34 1066:1066 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2770638Z STAGE:2022-12-01 10:36:34 1067:1067 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2770871Z STAGE:2022-12-01 10:36:34 1066:1066 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2771106Z STAGE:2022-12-01 10:36:34 1067:1067 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2771346Z STAGE:2022-12-01 10:36:34 1067:1067 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2771584Z STAGE:2022-12-01 10:36:34 1066:1066 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2771818Z STAGE:2022-12-01 10:36:34 1067:1067 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2772052Z STAGE:2022-12-01 10:36:34 1066:1066 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2772291Z STAGE:2022-12-01 10:36:34 1066:1066 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2772560Z STAGE:2022-12-01 10:36:34 1067:1067 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2772785Z STAGE:2022-12-01 10:36:34 1066:1066 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2773047Z STAGE:2022-12-01 10:36:34 1067:1067 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.2773365Z STAGE:2022-12-01 10:36:34 1067:1067 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2773611Z STAGE:2022-12-01 10:36:34 1066:1066 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.2773680Z ok (1.856s)
2022-12-01T10:43:19.2773685Z 
2022-12-01T10:43:19.2773882Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2773958Z Ran 1 test in 1.856s
2022-12-01T10:43:19.2773963Z 
2022-12-01T10:43:19.2774023Z OK
2022-12-01T10:43:19.2774028Z 
2022-12-01T10:43:19.2774099Z Generating XML reports...
2022-12-01T10:43:19.2774449Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103633.xml
2022-12-01T10:43:19.2774679Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.2774956Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2775081Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2775362Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2775497Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2775502Z 
2022-12-01T10:43:19.2775575Z Running tests...
2022-12-01T10:43:19.2775773Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2775995Z   test_broadcast_multigpu (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1126
2022-12-01T10:43:19.2776152Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1127
2022-12-01T10:43:19.2776306Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1128
2022-12-01T10:43:19.2776577Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2776700Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2776979Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2777114Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2777295Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:43:19.2777564Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2777677Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2777958Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2778093Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2778270Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:43:19.2778566Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2778836Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2778958Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2779232Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2779353Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2779570Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:43:19.2779940Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2780234Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2780512Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:43:19.2780678Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:43:19.2780842Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:43:19.2780938Z skip: CUDA is not available. (1.754s)
2022-12-01T10:43:19.2780945Z 
2022-12-01T10:43:19.2781142Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2781205Z Ran 1 test in 1.755s
2022-12-01T10:43:19.2781210Z 
2022-12-01T10:43:19.2781283Z OK (skipped=1)
2022-12-01T10:43:19.2781287Z 
2022-12-01T10:43:19.2781371Z Generating XML reports...
2022-12-01T10:43:19.2781713Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103636.xml
2022-12-01T10:43:19.2781946Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.2782220Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2782343Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2782624Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2782758Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2782763Z 
2022-12-01T10:43:19.2782822Z Running tests...
2022-12-01T10:43:19.2783020Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2783711Z   test_broadcast_object_list (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/82847 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.249s)
2022-12-01T10:43:19.2783717Z 
2022-12-01T10:43:19.2783912Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2783986Z Ran 1 test in 0.250s
2022-12-01T10:43:19.2783991Z 
2022-12-01T10:43:19.2784061Z OK (skipped=1)
2022-12-01T10:43:19.2784065Z 
2022-12-01T10:43:19.2784148Z Generating XML reports...
2022-12-01T10:43:19.2784484Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103639.xml
2022-12-01T10:43:19.2784716Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.2784989Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2785101Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2785380Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2785513Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2785518Z 
2022-12-01T10:43:19.2785588Z Running tests...
2022-12-01T10:43:19.2785779Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2786145Z   test_compute_bucket_assignment_by_size_sparse_error_with_logger (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'nccl', 'gloo'} (0.001s)
2022-12-01T10:43:19.2786150Z 
2022-12-01T10:43:19.2786394Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2786500Z Ran 1 test in 0.002s
2022-12-01T10:43:19.2786505Z 
2022-12-01T10:43:19.2786575Z OK (skipped=1)
2022-12-01T10:43:19.2786580Z 
2022-12-01T10:43:19.2786649Z Generating XML reports...
2022-12-01T10:43:19.2786987Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103641.xml
2022-12-01T10:43:19.2787220Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.2787494Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2787618Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2787902Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2788040Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2788045Z 
2022-12-01T10:43:19.2788119Z Running tests...
2022-12-01T10:43:19.2788301Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2788678Z   test_compute_bucket_assignment_by_size_sparse_error_without_logger (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'gloo', 'nccl'} (0.001s)
2022-12-01T10:43:19.2788683Z 
2022-12-01T10:43:19.2788876Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2788950Z Ran 1 test in 0.002s
2022-12-01T10:43:19.2788955Z 
2022-12-01T10:43:19.2789025Z OK (skipped=1)
2022-12-01T10:43:19.2789030Z 
2022-12-01T10:43:19.2789112Z Generating XML reports...
2022-12-01T10:43:19.2789451Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103642.xml
2022-12-01T10:43:19.2789684Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.2789962Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2790074Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2790353Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2790486Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2790491Z 
2022-12-01T10:43:19.2790562Z Running tests...
2022-12-01T10:43:19.2790755Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2790989Z   test_ddp_broadcast_buffer (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1209
2022-12-01T10:43:19.2791144Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1210
2022-12-01T10:43:19.2791314Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1211
2022-12-01T10:43:19.2791714Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2791827Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2792108Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2792242Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2792420Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:43:19.2792711Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2792832Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2793112Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2793312Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2793506Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:43:19.2793774Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2793895Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2794173Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2794308Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2794483Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:43:19.2794784Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2795086Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2795381Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2795532Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:43:19.2795698Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:43:19.2795860Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:43:19.2795964Z skip: Need at least 2 CUDA devices (1.754s)
2022-12-01T10:43:19.2795969Z 
2022-12-01T10:43:19.2796166Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2796241Z Ran 1 test in 1.754s
2022-12-01T10:43:19.2796246Z 
2022-12-01T10:43:19.2796316Z OK (skipped=1)
2022-12-01T10:43:19.2796320Z 
2022-12-01T10:43:19.2796405Z Generating XML reports...
2022-12-01T10:43:19.2796745Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103644.xml
2022-12-01T10:43:19.2796967Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.2797242Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2797365Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2797647Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2797782Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2797787Z 
2022-12-01T10:43:19.2797858Z Running tests...
2022-12-01T10:43:19.2798051Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2798298Z   test_ddp_broadcast_buffer_via_hook (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1262
2022-12-01T10:43:19.2798454Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1263
2022-12-01T10:43:19.2798595Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1264
2022-12-01T10:43:19.2798867Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2798988Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2799271Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2799405Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2799582Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:43:19.2799883Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2800033Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2800300Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2800434Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2800611Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:43:19.2800881Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2801002Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2801281Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2801417Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2801594Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:43:19.2801894Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2802235Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2802622Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2802788Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:43:19.2802947Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:43:19.2803109Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:43:19.2803216Z skip: Need at least 2 CUDA devices (1.854s)
2022-12-01T10:43:19.2803222Z 
2022-12-01T10:43:19.2803419Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2803493Z Ran 1 test in 1.855s
2022-12-01T10:43:19.2803498Z 
2022-12-01T10:43:19.2803567Z OK (skipped=1)
2022-12-01T10:43:19.2803572Z 
2022-12-01T10:43:19.2803641Z Generating XML reports...
2022-12-01T10:43:19.2803981Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103647.xml
2022-12-01T10:43:19.2804212Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.2804486Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2804607Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2804887Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2805022Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2805029Z 
2022-12-01T10:43:19.2805100Z Running tests...
2022-12-01T10:43:19.2805282Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2805976Z   test_ddp_buffer_hook_allreduce (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78641 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.237s)
2022-12-01T10:43:19.2805993Z 
2022-12-01T10:43:19.2806172Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2806244Z Ran 1 test in 0.237s
2022-12-01T10:43:19.2806249Z 
2022-12-01T10:43:19.2806318Z OK (skipped=1)
2022-12-01T10:43:19.2806323Z 
2022-12-01T10:43:19.2806404Z Generating XML reports...
2022-12-01T10:43:19.2806784Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103650.xml
2022-12-01T10:43:19.2807045Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.2807320Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2807443Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2807708Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2807842Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2807847Z 
2022-12-01T10:43:19.2807917Z Running tests...
2022-12-01T10:43:19.2808109Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2808824Z   test_ddp_buffer_hook_allreduce_return_future (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77261 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.236s)
2022-12-01T10:43:19.2808832Z 
2022-12-01T10:43:19.2809025Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2809098Z Ran 1 test in 0.236s
2022-12-01T10:43:19.2809103Z 
2022-12-01T10:43:19.2809173Z OK (skipped=1)
2022-12-01T10:43:19.2809178Z 
2022-12-01T10:43:19.2809262Z Generating XML reports...
2022-12-01T10:43:19.2809604Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103652.xml
2022-12-01T10:43:19.2809826Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.2810101Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2810226Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2810505Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2810639Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2810643Z 
2022-12-01T10:43:19.2810715Z Running tests...
2022-12-01T10:43:19.2810908Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2811242Z   test_ddp_build_debug_param_to_name_mapping (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'nccl', 'gloo'} (0.003s)
2022-12-01T10:43:19.2811248Z 
2022-12-01T10:43:19.2811438Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2811500Z Ran 1 test in 0.003s
2022-12-01T10:43:19.2811516Z 
2022-12-01T10:43:19.2811575Z OK (skipped=1)
2022-12-01T10:43:19.2811582Z 
2022-12-01T10:43:19.2811665Z Generating XML reports...
2022-12-01T10:43:19.2812001Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103654.xml
2022-12-01T10:43:19.2812233Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.2812504Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2812627Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2812907Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2813040Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2813045Z 
2022-12-01T10:43:19.2813104Z Running tests...
2022-12-01T10:43:19.2813364Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2813667Z   test_ddp_build_debug_param_to_name_mapping_requires_grad (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1345
2022-12-01T10:43:19.2813849Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1346
2022-12-01T10:43:19.2814000Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1347
2022-12-01T10:43:19.2814277Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2814400Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2814682Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2814816Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2814984Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:43:19.2815255Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2815379Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2815654Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2815786Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2815963Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:43:19.2816260Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2816526Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2816635Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2816916Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2817053Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2817227Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:43:19.2817521Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2817816Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2817981Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:43:19.2818143Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:43:19.2818308Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:43:19.2818399Z skip: Need at least 2 CUDA devices (1.754s)
2022-12-01T10:43:19.2818406Z 
2022-12-01T10:43:19.2818599Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2818672Z Ran 1 test in 1.754s
2022-12-01T10:43:19.2818678Z 
2022-12-01T10:43:19.2818747Z OK (skipped=1)
2022-12-01T10:43:19.2818752Z 
2022-12-01T10:43:19.2818834Z Generating XML reports...
2022-12-01T10:43:19.2819170Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103655.xml
2022-12-01T10:43:19.2819402Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.2819674Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2819846Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2820147Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2820310Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2820316Z 
2022-12-01T10:43:19.2820492Z Running tests...
2022-12-01T10:43:19.2820689Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2820931Z   test_ddp_comm_hook_logging (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1398
2022-12-01T10:43:19.2821086Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1399
2022-12-01T10:43:19.2821238Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1400
2022-12-01T10:43:19.2821507Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2821617Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2821899Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2822035Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2822208Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:43:19.2822479Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2822599Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2822879Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2823012Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2823189Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:43:19.2823448Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2823571Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2823847Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2823981Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2824155Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:43:19.2824452Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2824746Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2825034Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2825198Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:43:19.2825346Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:43:19.2825507Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:43:19.2825610Z skip: Need at least 3 CUDA devices (1.857s)
2022-12-01T10:43:19.2825615Z 
2022-12-01T10:43:19.2825810Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2825886Z Ran 1 test in 1.858s
2022-12-01T10:43:19.2825891Z 
2022-12-01T10:43:19.2825962Z OK (skipped=1)
2022-12-01T10:43:19.2825967Z 
2022-12-01T10:43:19.2826050Z Generating XML reports...
2022-12-01T10:43:19.2826393Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103658.xml
2022-12-01T10:43:19.2826612Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.2826938Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2827110Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2827389Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2827525Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2827530Z 
2022-12-01T10:43:19.2827603Z Running tests...
2022-12-01T10:43:19.2827802Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2828145Z   test_ddp_control_flow_different_across_ranks (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'gloo', 'nccl'} (0.004s)
2022-12-01T10:43:19.2828151Z 
2022-12-01T10:43:19.2828343Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2828406Z Ran 1 test in 0.004s
2022-12-01T10:43:19.2828426Z 
2022-12-01T10:43:19.2828485Z OK (skipped=1)
2022-12-01T10:43:19.2828492Z 
2022-12-01T10:43:19.2828576Z Generating XML reports...
2022-12-01T10:43:19.2828916Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103701.xml
2022-12-01T10:43:19.2829148Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.2829424Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2829548Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2829832Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2829966Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2829970Z 
2022-12-01T10:43:19.2830029Z Running tests...
2022-12-01T10:43:19.2830225Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2830562Z   test_ddp_control_flow_same_across_ranks (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'nccl', 'gloo'} (0.003s)
2022-12-01T10:43:19.2830567Z 
2022-12-01T10:43:19.2830759Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2830833Z Ran 1 test in 0.003s
2022-12-01T10:43:19.2830838Z 
2022-12-01T10:43:19.2830908Z OK (skipped=1)
2022-12-01T10:43:19.2830913Z 
2022-12-01T10:43:19.2830998Z Generating XML reports...
2022-12-01T10:43:19.2831334Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103703.xml
2022-12-01T10:43:19.2831564Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.2831823Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2831946Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2832228Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2832362Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2832367Z 
2022-12-01T10:43:19.2832439Z Running tests...
2022-12-01T10:43:19.2832631Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2832862Z   test_ddp_create_graph (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1471
2022-12-01T10:43:19.2833017Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1472
2022-12-01T10:43:19.2833168Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1473
2022-12-01T10:43:19.2833459Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2833583Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2833891Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2834026Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2834203Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:43:19.2834475Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2834597Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2834877Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2834998Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2835176Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:43:19.2835445Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2835567Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2835844Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2835978Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2836151Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:43:19.2836448Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2836742Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2837026Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2837193Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:43:19.2837357Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:43:19.2837542Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8ggr2dl5
2022-12-01T10:43:19.2837727Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp_9bde8e
2022-12-01T10:43:19.2837922Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8ggr2dl5/_remote_module_non_scriptable.py
2022-12-01T10:43:19.2838112Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp_9bde8e/_remote_module_non_scriptable.py
2022-12-01T10:43:19.2838273Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:43:19.2838456Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4h5c9o65
2022-12-01T10:43:19.2838638Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4h5c9o65/_remote_module_non_scriptable.py
2022-12-01T10:43:19.2839329Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-12-01T10:43:19.2840017Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-12-01T10:43:19.2840723Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-12-01T10:43:19.2841646Z /opt/conda/lib/python3.7/site-packages/torch/autograd/__init__.py:199: UserWarning: Using backward() with create_graph=True will create a reference cycle between the parameter and its gradient which can cause a memory leak. We recommend using autograd.grad when creating the graph to avoid this. If you have to use this function, make sure to reset the .grad fields of your parameters to None after use to break the cycle and avoid the leak. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/engine.cpp:1064.)
2022-12-01T10:43:19.2841818Z   allow_unreachable=True, accumulate_grad=True)  # Calls into the C++ engine to run the backward pass
2022-12-01T10:43:19.2841989Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:43:19.2842870Z /opt/conda/lib/python3.7/site-packages/torch/autograd/__init__.py:199: UserWarning: Using backward() with create_graph=True will create a reference cycle between the parameter and its gradient which can cause a memory leak. We recommend using autograd.grad when creating the graph to avoid this. If you have to use this function, make sure to reset the .grad fields of your parameters to None after use to break the cycle and avoid the leak. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/engine.cpp:1064.)
2022-12-01T10:43:19.2843039Z   allow_unreachable=True, accumulate_grad=True)  # Calls into the C++ engine to run the backward pass
2022-12-01T10:43:19.2843205Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:43:19.2844079Z /opt/conda/lib/python3.7/site-packages/torch/autograd/__init__.py:199: UserWarning: Using backward() with create_graph=True will create a reference cycle between the parameter and its gradient which can cause a memory leak. We recommend using autograd.grad when creating the graph to avoid this. If you have to use this function, make sure to reset the .grad fields of your parameters to None after use to break the cycle and avoid the leak. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/engine.cpp:1064.)
2022-12-01T10:43:19.2844247Z   allow_unreachable=True, accumulate_grad=True)  # Calls into the C++ engine to run the backward pass
2022-12-01T10:43:19.2844416Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:43:19.2845101Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-12-01T10:43:19.2845781Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-12-01T10:43:19.2846462Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-12-01T10:43:19.2847169Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-12-01T10:43:19.2847871Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-12-01T10:43:19.2848547Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-12-01T10:43:19.2849217Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-12-01T10:43:19.2849885Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-12-01T10:43:19.2850552Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-12-01T10:43:19.2851230Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-12-01T10:43:19.2851903Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-12-01T10:43:19.2852570Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-12-01T10:43:19.2853235Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-12-01T10:43:19.2854025Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-12-01T10:43:19.2854722Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-12-01T10:43:19.2854793Z ok (1.859s)
2022-12-01T10:43:19.2854798Z 
2022-12-01T10:43:19.2854995Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2855071Z Ran 1 test in 1.859s
2022-12-01T10:43:19.2855076Z 
2022-12-01T10:43:19.2855136Z OK
2022-12-01T10:43:19.2855140Z 
2022-12-01T10:43:19.2855223Z Generating XML reports...
2022-12-01T10:43:19.2855567Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103704.xml
2022-12-01T10:43:19.2855790Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.2856063Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2856186Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2856467Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2856604Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2856609Z 
2022-12-01T10:43:19.2856681Z Running tests...
2022-12-01T10:43:19.2856878Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2857175Z   test_ddp_device (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'nccl', 'gloo'} (0.004s)
2022-12-01T10:43:19.2857180Z 
2022-12-01T10:43:19.2857376Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2857440Z Ran 1 test in 0.004s
2022-12-01T10:43:19.2857456Z 
2022-12-01T10:43:19.2857514Z OK (skipped=1)
2022-12-01T10:43:19.2857518Z 
2022-12-01T10:43:19.2857604Z Generating XML reports...
2022-12-01T10:43:19.2857946Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103707.xml
2022-12-01T10:43:19.2858177Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.2858449Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2858572Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2858852Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2858989Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2858996Z 
2022-12-01T10:43:19.2859055Z Running tests...
2022-12-01T10:43:19.2859248Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2859563Z   test_ddp_forward_backward_hook (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'nccl', 'gloo'} (0.003s)
2022-12-01T10:43:19.2859568Z 
2022-12-01T10:43:19.2859759Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2859837Z Ran 1 test in 0.003s
2022-12-01T10:43:19.2859842Z 
2022-12-01T10:43:19.2859912Z OK (skipped=1)
2022-12-01T10:43:19.2859917Z 
2022-12-01T10:43:19.2859998Z Generating XML reports...
2022-12-01T10:43:19.2860334Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103709.xml
2022-12-01T10:43:19.2860673Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.2860985Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2861140Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2861419Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2861553Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2861558Z 
2022-12-01T10:43:19.2861631Z Running tests...
2022-12-01T10:43:19.2861825Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2862520Z   test_ddp_grad_div_uneven_inputs (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78685 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.237s)
2022-12-01T10:43:19.2862526Z 
2022-12-01T10:43:19.2862724Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2862798Z Ran 1 test in 0.237s
2022-12-01T10:43:19.2862803Z 
2022-12-01T10:43:19.2862874Z OK (skipped=1)
2022-12-01T10:43:19.2862879Z 
2022-12-01T10:43:19.2862949Z Generating XML reports...
2022-12-01T10:43:19.2863287Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103710.xml
2022-12-01T10:43:19.2863520Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.2863792Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2863915Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2864197Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2864332Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2864340Z 
2022-12-01T10:43:19.2864411Z Running tests...
2022-12-01T10:43:19.2864595Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2865278Z   test_ddp_hook_parity_allreduce (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77293 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.234s)
2022-12-01T10:43:19.2865297Z 
2022-12-01T10:43:19.2865476Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2865550Z Ran 1 test in 0.234s
2022-12-01T10:43:19.2865556Z 
2022-12-01T10:43:19.2865625Z OK (skipped=1)
2022-12-01T10:43:19.2865652Z 
2022-12-01T10:43:19.2865737Z Generating XML reports...
2022-12-01T10:43:19.2866077Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103712.xml
2022-12-01T10:43:19.2866314Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.2866587Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2866709Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2866977Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2867111Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2867116Z 
2022-12-01T10:43:19.2867188Z Running tests...
2022-12-01T10:43:19.2867380Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2867662Z   test_ddp_hook_parity_allreduce_process_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1564
2022-12-01T10:43:19.2867842Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1565
2022-12-01T10:43:19.2867994Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1566
2022-12-01T10:43:19.2868265Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2868386Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2868652Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2868788Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2868965Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:43:19.2869237Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2869361Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2869637Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2869772Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2869949Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:43:19.2870216Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2870327Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2870603Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2870739Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2870913Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:43:19.2871210Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2871507Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2871794Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2871957Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:43:19.2872108Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:43:19.2872270Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:43:19.2872373Z skip: Need at least 3 CUDA devices (1.859s)
2022-12-01T10:43:19.2872380Z 
2022-12-01T10:43:19.2872576Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2872652Z Ran 1 test in 1.859s
2022-12-01T10:43:19.2872657Z 
2022-12-01T10:43:19.2872726Z OK (skipped=1)
2022-12-01T10:43:19.2872731Z 
2022-12-01T10:43:19.2872815Z Generating XML reports...
2022-12-01T10:43:19.2873150Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103713.xml
2022-12-01T10:43:19.2873379Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.2873637Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2873759Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2874035Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2874198Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2874238Z 
2022-12-01T10:43:19.2874309Z Running tests...
2022-12-01T10:43:19.2874502Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2874743Z   test_ddp_hook_parity_post_localSGD (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1617
2022-12-01T10:43:19.2874897Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1618
2022-12-01T10:43:19.2875048Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1619
2022-12-01T10:43:19.2875304Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2875425Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2875701Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2875835Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2876015Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:43:19.2876280Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2876400Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2876677Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2876799Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2876973Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:43:19.2877239Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2877361Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2877640Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2877773Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2877946Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:43:19.2878240Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2878536Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2878812Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2878979Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:43:19.2879134Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:43:19.2879296Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:43:19.2879401Z skip: Need at least 3 CUDA devices (1.851s)
2022-12-01T10:43:19.2879406Z 
2022-12-01T10:43:19.2879601Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2879674Z Ran 1 test in 1.852s
2022-12-01T10:43:19.2879679Z 
2022-12-01T10:43:19.2879749Z OK (skipped=1)
2022-12-01T10:43:19.2879753Z 
2022-12-01T10:43:19.2879836Z Generating XML reports...
2022-12-01T10:43:19.2880163Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103717.xml
2022-12-01T10:43:19.2880394Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.2880695Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2880851Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2881134Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2881270Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2881275Z 
2022-12-01T10:43:19.2881346Z Running tests...
2022-12-01T10:43:19.2881539Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2882229Z   test_ddp_hook_parity_powerSGD (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77378 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.237s)
2022-12-01T10:43:19.2882235Z 
2022-12-01T10:43:19.2882431Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2882494Z Ran 1 test in 0.237s
2022-12-01T10:43:19.2882499Z 
2022-12-01T10:43:19.2882567Z OK (skipped=1)
2022-12-01T10:43:19.2882571Z 
2022-12-01T10:43:19.2882652Z Generating XML reports...
2022-12-01T10:43:19.2882994Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103720.xml
2022-12-01T10:43:19.2883227Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.2883498Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2883620Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2883896Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2884019Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2884038Z 
2022-12-01T10:43:19.2884097Z Running tests...
2022-12-01T10:43:19.2884288Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2884527Z   test_ddp_hook_pickling_powerSGD (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1680
2022-12-01T10:43:19.2884682Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1681
2022-12-01T10:43:19.2884835Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1682
2022-12-01T10:43:19.2885104Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2885225Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2885503Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2885628Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2885807Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:43:19.2886074Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2886195Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2886468Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2886603Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2886779Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:43:19.2887046Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2887195Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2887464Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2887624Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2887798Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:43:19.2888096Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2888394Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2888684Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2888849Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:43:19.2889018Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:43:19.2889183Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:43:19.2889275Z skip: Need at least 3 CUDA devices (1.756s)
2022-12-01T10:43:19.2889281Z 
2022-12-01T10:43:19.2889475Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2889551Z Ran 1 test in 1.756s
2022-12-01T10:43:19.2889556Z 
2022-12-01T10:43:19.2889629Z OK (skipped=1)
2022-12-01T10:43:19.2889634Z 
2022-12-01T10:43:19.2889718Z Generating XML reports...
2022-12-01T10:43:19.2890062Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103722.xml
2022-12-01T10:43:19.2890296Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.2890570Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2890683Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2890964Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2891099Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2891103Z 
2022-12-01T10:43:19.2891176Z Running tests...
2022-12-01T10:43:19.2891369Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2891642Z   test_ddp_hook_with_optimizer_parity_adam_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1733
2022-12-01T10:43:19.2891798Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1734
2022-12-01T10:43:19.2891951Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1735
2022-12-01T10:43:19.2892223Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2892336Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2892618Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2892750Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2892926Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:43:19.2893196Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2893397Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2893681Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2893850Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2894016Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:43:19.2894350Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2894473Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2894750Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2894883Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2895058Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:43:19.2895359Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2895659Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2895953Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2896105Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:43:19.2896264Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:43:19.2896426Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:43:19.2896529Z skip: Need at least 2 CUDA devices (1.757s)
2022-12-01T10:43:19.2896534Z 
2022-12-01T10:43:19.2896728Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2896802Z Ran 1 test in 1.757s
2022-12-01T10:43:19.2896807Z 
2022-12-01T10:43:19.2896879Z OK (skipped=1)
2022-12-01T10:43:19.2896884Z 
2022-12-01T10:43:19.2896969Z Generating XML reports...
2022-12-01T10:43:19.2897309Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103725.xml
2022-12-01T10:43:19.2897527Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.2897802Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2897925Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2898203Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2898337Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2898342Z 
2022-12-01T10:43:19.2898415Z Running tests...
2022-12-01T10:43:19.2898608Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2898877Z   test_ddp_hook_with_optimizer_parity_adam_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1786
2022-12-01T10:43:19.2899033Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1787
2022-12-01T10:43:19.2899174Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1788
2022-12-01T10:43:19.2899442Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2899565Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2899841Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2899974Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2900149Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:43:19.2900596Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2900722Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2901026Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2901162Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2901339Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:43:19.2901609Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2901731Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2902009Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2902143Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2902321Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:43:19.2902624Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2902906Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2903193Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2903355Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:43:19.2903513Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:43:19.2903672Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:43:19.2903775Z skip: Need at least 2 CUDA devices (1.752s)
2022-12-01T10:43:19.2903780Z 
2022-12-01T10:43:19.2903977Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2904053Z Ran 1 test in 1.752s
2022-12-01T10:43:19.2904058Z 
2022-12-01T10:43:19.2904129Z OK (skipped=1)
2022-12-01T10:43:19.2904133Z 
2022-12-01T10:43:19.2904204Z Generating XML reports...
2022-12-01T10:43:19.2904545Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103728.xml
2022-12-01T10:43:19.2904777Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.2905048Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2905172Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2905448Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2905583Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2905590Z 
2022-12-01T10:43:19.2905662Z Running tests...
2022-12-01T10:43:19.2905844Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2906158Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1839
2022-12-01T10:43:19.2906316Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1840
2022-12-01T10:43:19.2906467Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1841
2022-12-01T10:43:19.2906739Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2906863Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2907172Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2907335Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2907511Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:43:19.2907781Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2907891Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2908168Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2908304Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2908479Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:43:19.2908750Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2908872Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2909150Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2909283Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2909446Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:43:19.2909743Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2910040Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2910326Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2910492Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:43:19.2910653Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:43:19.2910814Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:43:19.2910916Z skip: Need at least 2 CUDA devices (1.751s)
2022-12-01T10:43:19.2910922Z 
2022-12-01T10:43:19.2911117Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2911179Z Ran 1 test in 1.751s
2022-12-01T10:43:19.2911185Z 
2022-12-01T10:43:19.2911255Z OK (skipped=1)
2022-12-01T10:43:19.2911259Z 
2022-12-01T10:43:19.2911341Z Generating XML reports...
2022-12-01T10:43:19.2911679Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103731.xml
2022-12-01T10:43:19.2911908Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.2912181Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2912306Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2912584Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2912717Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2912722Z 
2022-12-01T10:43:19.2912783Z Running tests...
2022-12-01T10:43:19.2912977Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2913287Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1892
2022-12-01T10:43:19.2913440Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1893
2022-12-01T10:43:19.2913643Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1894
2022-12-01T10:43:19.2913941Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2914064Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2914343Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2914480Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2914645Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:43:19.2914913Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2915034Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2915316Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2915453Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2915629Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:43:19.2915898Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2916020Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2916283Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2916417Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2916591Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:43:19.2916891Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2917188Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2917482Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2917646Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:43:19.2917807Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:43:19.2917966Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:43:19.2918056Z skip: Need at least 2 CUDA devices (1.752s)
2022-12-01T10:43:19.2918073Z 
2022-12-01T10:43:19.2918255Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2918329Z Ran 1 test in 1.752s
2022-12-01T10:43:19.2918334Z 
2022-12-01T10:43:19.2918404Z OK (skipped=1)
2022-12-01T10:43:19.2918410Z 
2022-12-01T10:43:19.2918493Z Generating XML reports...
2022-12-01T10:43:19.2918835Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103735.xml
2022-12-01T10:43:19.2919065Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.2919334Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2919456Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2919721Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2919854Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2919859Z 
2022-12-01T10:43:19.2919931Z Running tests...
2022-12-01T10:43:19.2920123Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2920462Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1945
2022-12-01T10:43:19.2920652Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1946
2022-12-01T10:43:19.2920804Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1947
2022-12-01T10:43:19.2921073Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2921195Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2921459Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2921592Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2921770Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:43:19.2922040Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2922163Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2922442Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2922575Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2922752Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:43:19.2923007Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2923128Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2923407Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2923539Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2923716Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:43:19.2924013Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2924312Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2924598Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2924763Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:43:19.2924911Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:43:19.2925075Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:43:19.2925181Z skip: Need at least 2 CUDA devices (1.853s)
2022-12-01T10:43:19.2925187Z 
2022-12-01T10:43:19.2925383Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2925457Z Ran 1 test in 1.853s
2022-12-01T10:43:19.2925462Z 
2022-12-01T10:43:19.2925532Z OK (skipped=1)
2022-12-01T10:43:19.2925537Z 
2022-12-01T10:43:19.2925621Z Generating XML reports...
2022-12-01T10:43:19.2925965Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103738.xml
2022-12-01T10:43:19.2926195Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.2926454Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2926575Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2926892Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2927053Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2927058Z 
2022-12-01T10:43:19.2927132Z Running tests...
2022-12-01T10:43:19.2927322Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2927635Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1998
2022-12-01T10:43:19.2927789Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1999
2022-12-01T10:43:19.2927939Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2000
2022-12-01T10:43:19.2928197Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2928320Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2928601Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2928735Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2928909Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:43:19.2929179Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2929301Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2929579Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2929700Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2929878Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:43:19.2930144Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2930269Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2930541Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2930673Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2930846Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:43:19.2931141Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2931438Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2931718Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2931884Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:43:19.2932043Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:43:19.2932205Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:43:19.2932308Z skip: Need at least 2 CUDA devices (1.754s)
2022-12-01T10:43:19.2932313Z 
2022-12-01T10:43:19.2932508Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2932582Z Ran 1 test in 1.754s
2022-12-01T10:43:19.2932588Z 
2022-12-01T10:43:19.2932658Z OK (skipped=1)
2022-12-01T10:43:19.2932663Z 
2022-12-01T10:43:19.2932745Z Generating XML reports...
2022-12-01T10:43:19.2933072Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103741.xml
2022-12-01T10:43:19.2933422Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.2933731Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2933856Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2934135Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2934270Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2934275Z 
2022-12-01T10:43:19.2934347Z Running tests...
2022-12-01T10:43:19.2934541Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2934857Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2051
2022-12-01T10:43:19.2935000Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2052
2022-12-01T10:43:19.2935155Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2053
2022-12-01T10:43:19.2935424Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2935546Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2935825Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2935959Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2936137Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:43:19.2936405Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2936530Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2936798Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2936932Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2937107Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:43:19.2937375Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2937497Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2937770Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2937902Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2938079Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:43:19.2938365Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2938663Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2939006Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2939169Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:43:19.2939327Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:43:19.2939491Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:43:19.2939594Z skip: Need at least 2 CUDA devices (1.754s)
2022-12-01T10:43:19.2939599Z 
2022-12-01T10:43:19.2939825Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2939901Z Ran 1 test in 1.755s
2022-12-01T10:43:19.2939935Z 
2022-12-01T10:43:19.2939993Z OK (skipped=1)
2022-12-01T10:43:19.2939998Z 
2022-12-01T10:43:19.2940081Z Generating XML reports...
2022-12-01T10:43:19.2940522Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103744.xml
2022-12-01T10:43:19.2940754Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.2941028Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2941151Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2941430Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2941564Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2941569Z 
2022-12-01T10:43:19.2941644Z Running tests...
2022-12-01T10:43:19.2941827Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2942138Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2104
2022-12-01T10:43:19.2942292Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2105
2022-12-01T10:43:19.2942445Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2106
2022-12-01T10:43:19.2942715Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2942839Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2943119Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2943255Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2943436Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:43:19.2943696Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2943818Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2944095Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2944229Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2944405Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:43:19.2944672Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2944796Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2945074Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2945194Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2945369Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:43:19.2945661Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2945956Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2946247Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2946410Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:43:19.2946616Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:43:19.2946812Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:43:19.2946917Z skip: Need at least 2 CUDA devices (1.853s)
2022-12-01T10:43:19.2946923Z 
2022-12-01T10:43:19.2947103Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2947177Z Ran 1 test in 1.853s
2022-12-01T10:43:19.2947183Z 
2022-12-01T10:43:19.2947252Z OK (skipped=1)
2022-12-01T10:43:19.2947257Z 
2022-12-01T10:43:19.2947339Z Generating XML reports...
2022-12-01T10:43:19.2947674Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103748.xml
2022-12-01T10:43:19.2947903Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.2948179Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2948302Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2948582Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2948702Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2948708Z 
2022-12-01T10:43:19.2948781Z Running tests...
2022-12-01T10:43:19.2948975Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2949284Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2157
2022-12-01T10:43:19.2949436Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2158
2022-12-01T10:43:19.2949587Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2159
2022-12-01T10:43:19.2949859Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2949984Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2950260Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2950383Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2950560Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:43:19.2950830Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2950954Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2951231Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2951367Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2951549Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:43:19.2951817Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2951928Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2952203Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2952336Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2952512Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:43:19.2952809Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2953136Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2953454Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2953622Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:43:19.2953782Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:43:19.2953932Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:43:19.2954038Z skip: Need at least 2 CUDA devices (1.854s)
2022-12-01T10:43:19.2954043Z 
2022-12-01T10:43:19.2954238Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2954311Z Ran 1 test in 1.855s
2022-12-01T10:43:19.2954316Z 
2022-12-01T10:43:19.2954385Z OK (skipped=1)
2022-12-01T10:43:19.2954389Z 
2022-12-01T10:43:19.2954471Z Generating XML reports...
2022-12-01T10:43:19.2954808Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103751.xml
2022-12-01T10:43:19.2955039Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.2955311Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2955421Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2955697Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2955831Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2955836Z 
2022-12-01T10:43:19.2955907Z Running tests...
2022-12-01T10:43:19.2956098Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2956411Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2210
2022-12-01T10:43:19.2956566Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2211
2022-12-01T10:43:19.2956718Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2212
2022-12-01T10:43:19.2956988Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2957097Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2957374Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2957511Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2957688Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:43:19.2957959Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2958083Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2958359Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2958491Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2958655Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:43:19.2958922Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2959044Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2959322Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2959485Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2959689Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:43:19.2959987Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2960282Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2960572Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2960724Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:43:19.2960882Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:43:19.2961043Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:43:19.2961148Z skip: Need at least 2 CUDA devices (1.753s)
2022-12-01T10:43:19.2961155Z 
2022-12-01T10:43:19.2961349Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2961423Z Ran 1 test in 1.753s
2022-12-01T10:43:19.2961428Z 
2022-12-01T10:43:19.2961498Z OK (skipped=1)
2022-12-01T10:43:19.2961503Z 
2022-12-01T10:43:19.2961585Z Generating XML reports...
2022-12-01T10:43:19.2961922Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103754.xml
2022-12-01T10:43:19.2962142Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.2962413Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2962534Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2962814Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2962948Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2962955Z 
2022-12-01T10:43:19.2963026Z Running tests...
2022-12-01T10:43:19.2963218Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2963489Z   test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2263
2022-12-01T10:43:19.2963644Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2264
2022-12-01T10:43:19.2963783Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2265
2022-12-01T10:43:19.2964054Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2964177Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2964460Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2964596Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2964773Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:43:19.2965040Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2965163Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2965427Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2965561Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2965734Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:43:19.2966033Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2966195Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2966470Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2966604Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2966778Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:43:19.2967075Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2967360Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2967650Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2967815Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:43:19.2967975Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:43:19.2968137Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:43:19.2968239Z skip: Need at least 2 CUDA devices (1.854s)
2022-12-01T10:43:19.2968244Z 
2022-12-01T10:43:19.2968439Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2968512Z Ran 1 test in 1.854s
2022-12-01T10:43:19.2968518Z 
2022-12-01T10:43:19.2968588Z OK (skipped=1)
2022-12-01T10:43:19.2968593Z 
2022-12-01T10:43:19.2968664Z Generating XML reports...
2022-12-01T10:43:19.2969002Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103757.xml
2022-12-01T10:43:19.2969233Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.2969506Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2969629Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2969905Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2970039Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2970044Z 
2022-12-01T10:43:19.2970115Z Running tests...
2022-12-01T10:43:19.2970293Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2970562Z   test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2316
2022-12-01T10:43:19.2970715Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2317
2022-12-01T10:43:19.2970869Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2318
2022-12-01T10:43:19.2971145Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2971266Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2971543Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2971676Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2971852Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:43:19.2972109Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2972231Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2972539Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2972701Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2972878Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:43:19.2973148Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2973350Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2973632Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2973767Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2973930Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:43:19.2974232Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2974527Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2974824Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2974989Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:43:19.2975148Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:43:19.2975311Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:43:19.2975416Z skip: Need at least 2 CUDA devices (1.858s)
2022-12-01T10:43:19.2975421Z 
2022-12-01T10:43:19.2975616Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2975678Z Ran 1 test in 1.858s
2022-12-01T10:43:19.2975683Z 
2022-12-01T10:43:19.2975756Z OK (skipped=1)
2022-12-01T10:43:19.2975762Z 
2022-12-01T10:43:19.2975845Z Generating XML reports...
2022-12-01T10:43:19.2976184Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103801.xml
2022-12-01T10:43:19.2976410Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.2976681Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2976802Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2977079Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2977202Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2977218Z 
2022-12-01T10:43:19.2977276Z Running tests...
2022-12-01T10:43:19.2977470Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2977786Z   test_ddp_ignore_params_arg (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'nccl', 'gloo'} (0.001s)
2022-12-01T10:43:19.2977793Z 
2022-12-01T10:43:19.2977984Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2978059Z Ran 1 test in 0.002s
2022-12-01T10:43:19.2978064Z 
2022-12-01T10:43:19.2978135Z OK (skipped=1)
2022-12-01T10:43:19.2978140Z 
2022-12-01T10:43:19.2978222Z Generating XML reports...
2022-12-01T10:43:19.2978561Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103804.xml
2022-12-01T10:43:19.2978780Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.2979051Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2979172Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2979487Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2979646Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2979651Z 
2022-12-01T10:43:19.2979723Z Running tests...
2022-12-01T10:43:19.2979918Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2980146Z   test_ddp_inference (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2379
2022-12-01T10:43:19.2980302Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2380
2022-12-01T10:43:19.2980571Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2381
2022-12-01T10:43:19.2980845Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2980971Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2981250Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2981387Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2981563Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:43:19.2981836Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2981959Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2982225Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2982357Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2982533Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:43:19.2982810Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2982935Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2983214Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2983346Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2983523Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:43:19.2983822Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2984106Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2984405Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2984570Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:43:19.2984741Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:43:19.2984901Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:43:19.2985004Z skip: Need at least 2 CUDA devices (1.855s)
2022-12-01T10:43:19.2985009Z 
2022-12-01T10:43:19.2985204Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2985278Z Ran 1 test in 1.855s
2022-12-01T10:43:19.2985283Z 
2022-12-01T10:43:19.2985352Z OK (skipped=1)
2022-12-01T10:43:19.2985356Z 
2022-12-01T10:43:19.2985426Z Generating XML reports...
2022-12-01T10:43:19.2985765Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103805.xml
2022-12-01T10:43:19.2986040Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.2986348Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2986472Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2986752Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2986887Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2986894Z 
2022-12-01T10:43:19.2986967Z Running tests...
2022-12-01T10:43:19.2987160Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2987390Z   test_ddp_join_model_equivalence (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2432
2022-12-01T10:43:19.2987545Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2433
2022-12-01T10:43:19.2987698Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2434
2022-12-01T10:43:19.2987971Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2988095Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2988374Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2988508Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2988684Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:43:19.2988942Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2989064Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2989344Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2989481Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2989658Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:43:19.2989928Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2990051Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2990327Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2990460Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2990622Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:43:19.2990919Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2991213Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2991500Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2991667Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:43:19.2991832Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:43:19.2991991Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:43:19.2992095Z skip: Need at least 2 CUDA devices (1.754s)
2022-12-01T10:43:19.2992100Z 
2022-12-01T10:43:19.2992294Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2992355Z Ran 1 test in 1.754s
2022-12-01T10:43:19.2992360Z 
2022-12-01T10:43:19.2992459Z OK (skipped=1)
2022-12-01T10:43:19.2992464Z 
2022-12-01T10:43:19.2992576Z Generating XML reports...
2022-12-01T10:43:19.2992993Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103809.xml
2022-12-01T10:43:19.2993227Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.2993500Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2993624Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2993900Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2994021Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2994038Z 
2022-12-01T10:43:19.2994098Z Running tests...
2022-12-01T10:43:19.2994293Z ----------------------------------------------------------------------
2022-12-01T10:43:19.2994529Z   test_ddp_logging_data_cpu (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2485
2022-12-01T10:43:19.2994686Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2486
2022-12-01T10:43:19.2994839Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2487
2022-12-01T10:43:19.2995106Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2995229Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2995504Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2995627Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2995803Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:43:19.2996073Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2996197Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2996474Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2996608Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2996783Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:43:19.2997078Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2997345Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.2997454Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.2997735Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.2997868Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.2998043Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:43:19.2998335Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2998628Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.2998792Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:43:19.2998952Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:43:19.2999108Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:43:19.2999324Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprf4pu9_1
2022-12-01T10:43:19.2999541Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqbfi2r6h
2022-12-01T10:43:19.2999717Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_4_24fwj
2022-12-01T10:43:19.2999913Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprf4pu9_1/_remote_module_non_scriptable.py
2022-12-01T10:43:19.3000107Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqbfi2r6h/_remote_module_non_scriptable.py
2022-12-01T10:43:19.3000293Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_4_24fwj/_remote_module_non_scriptable.py
2022-12-01T10:43:19.3000463Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:43:19.3000630Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:43:19.3000786Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:43:19.3000855Z ok (1.959s)
2022-12-01T10:43:19.3000861Z 
2022-12-01T10:43:19.3001058Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3001131Z Ran 1 test in 1.959s
2022-12-01T10:43:19.3001136Z 
2022-12-01T10:43:19.3001195Z OK
2022-12-01T10:43:19.3001199Z 
2022-12-01T10:43:19.3001281Z Generating XML reports...
2022-12-01T10:43:19.3001621Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103812.xml
2022-12-01T10:43:19.3001854Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.3002131Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3002242Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3002523Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3002659Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3002664Z 
2022-12-01T10:43:19.3002736Z Running tests...
2022-12-01T10:43:19.3002931Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3003165Z   test_ddp_logging_data_gpu (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2547
2022-12-01T10:43:19.3003318Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2548
2022-12-01T10:43:19.3003469Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2549
2022-12-01T10:43:19.3003726Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3003850Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3004130Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3004265Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3004441Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:43:19.3004737Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.3005005Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3005127Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3005406Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3005525Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3005733Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:43:19.3006033Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3006155Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3006432Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3006565Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3006740Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:43:19.3007038Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.3007332Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.3007487Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:43:19.3007652Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:43:19.3007811Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:43:19.3007906Z skip: CUDA is not available. (1.854s)
2022-12-01T10:43:19.3007911Z 
2022-12-01T10:43:19.3008105Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3008178Z Ran 1 test in 1.855s
2022-12-01T10:43:19.3008183Z 
2022-12-01T10:43:19.3008252Z OK (skipped=1)
2022-12-01T10:43:19.3008257Z 
2022-12-01T10:43:19.3008342Z Generating XML reports...
2022-12-01T10:43:19.3008668Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103815.xml
2022-12-01T10:43:19.3008902Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.3009174Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3009299Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3009578Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3009713Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3009718Z 
2022-12-01T10:43:19.3009789Z Running tests...
2022-12-01T10:43:19.3009984Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3010322Z   test_ddp_model_diff_num_params_across_ranks (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'nccl', 'gloo'} (0.002s)
2022-12-01T10:43:19.3010328Z 
2022-12-01T10:43:19.3010507Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3010583Z Ran 1 test in 0.002s
2022-12-01T10:43:19.3010588Z 
2022-12-01T10:43:19.3010660Z OK (skipped=1)
2022-12-01T10:43:19.3010664Z 
2022-12-01T10:43:19.3010745Z Generating XML reports...
2022-12-01T10:43:19.3011079Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103819.xml
2022-12-01T10:43:19.3011308Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.3011578Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3011700Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3011976Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3012099Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3012104Z 
2022-12-01T10:43:19.3012175Z Running tests...
2022-12-01T10:43:19.3012397Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3012765Z   test_ddp_model_diff_shape_across_ranks (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'nccl', 'gloo'} (0.002s)
2022-12-01T10:43:19.3012771Z 
2022-12-01T10:43:19.3012963Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3013037Z Ran 1 test in 0.002s
2022-12-01T10:43:19.3013042Z 
2022-12-01T10:43:19.3013111Z OK (skipped=1)
2022-12-01T10:43:19.3013115Z 
2022-12-01T10:43:19.3013196Z Generating XML reports...
2022-12-01T10:43:19.3013643Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103820.xml
2022-12-01T10:43:19.3013864Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.3014143Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3014267Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3014550Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3014684Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3014690Z 
2022-12-01T10:43:19.3014764Z Running tests...
2022-12-01T10:43:19.3014956Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3015315Z   test_ddp_multiple_nested_unused_params_err_ignore_params (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'nccl', 'gloo'} (0.001s)
2022-12-01T10:43:19.3015321Z 
2022-12-01T10:43:19.3015513Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3015576Z Ran 1 test in 0.002s
2022-12-01T10:43:19.3015581Z 
2022-12-01T10:43:19.3015653Z OK (skipped=1)
2022-12-01T10:43:19.3015657Z 
2022-12-01T10:43:19.3015743Z Generating XML reports...
2022-12-01T10:43:19.3016077Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103821.xml
2022-12-01T10:43:19.3016314Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.3016585Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3016710Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3016991Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3017115Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3017132Z 
2022-12-01T10:43:19.3017192Z Running tests...
2022-12-01T10:43:19.3017383Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3017726Z   test_ddp_multiple_nested_unused_params_error (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'gloo', 'nccl'} (0.002s)
2022-12-01T10:43:19.3017733Z 
2022-12-01T10:43:19.3017923Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3017997Z Ran 1 test in 0.002s
2022-12-01T10:43:19.3018002Z 
2022-12-01T10:43:19.3018070Z OK (skipped=1)
2022-12-01T10:43:19.3018074Z 
2022-12-01T10:43:19.3018155Z Generating XML reports...
2022-12-01T10:43:19.3018491Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103823.xml
2022-12-01T10:43:19.3018709Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.3018984Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3019105Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3019422Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3019589Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3019594Z 
2022-12-01T10:43:19.3019665Z Running tests...
2022-12-01T10:43:19.3019859Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3020157Z   test_ddp_namedtuple (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'gloo', 'nccl'} (0.002s)
2022-12-01T10:43:19.3020162Z 
2022-12-01T10:43:19.3020465Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3020527Z Ran 1 test in 0.002s
2022-12-01T10:43:19.3020532Z 
2022-12-01T10:43:19.3020604Z OK (skipped=1)
2022-12-01T10:43:19.3020608Z 
2022-12-01T10:43:19.3020691Z Generating XML reports...
2022-12-01T10:43:19.3021029Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103824.xml
2022-12-01T10:43:19.3021261Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.3021541Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3021663Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3021942Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3022076Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3022081Z 
2022-12-01T10:43:19.3022140Z Running tests...
2022-12-01T10:43:19.3022333Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3022566Z   test_ddp_new_tensor_in_fwd (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2650
2022-12-01T10:43:19.3022723Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2651
2022-12-01T10:43:19.3022882Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2652
2022-12-01T10:43:19.3023152Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3023272Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3023551Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3023671Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3023848Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:43:19.3024118Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3024241Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3024516Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3024651Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3024829Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:43:19.3025095Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3025215Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3025478Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3025609Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3025782Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:43:19.3026128Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.3026464Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.3026754Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.3026921Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:43:19.3027084Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:43:19.3027245Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:43:19.3027336Z skip: Need at least 2 CUDA devices (1.753s)
2022-12-01T10:43:19.3027341Z 
2022-12-01T10:43:19.3027536Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3027613Z Ran 1 test in 1.754s
2022-12-01T10:43:19.3027619Z 
2022-12-01T10:43:19.3027691Z OK (skipped=1)
2022-12-01T10:43:19.3027695Z 
2022-12-01T10:43:19.3027780Z Generating XML reports...
2022-12-01T10:43:19.3028117Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103825.xml
2022-12-01T10:43:19.3028349Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.3028621Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3028744Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3029011Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3029144Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3029149Z 
2022-12-01T10:43:19.3029221Z Running tests...
2022-12-01T10:43:19.3029422Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3030133Z   test_ddp_new_tensor_in_fwd_static_graph (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78338 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.237s)
2022-12-01T10:43:19.3030139Z 
2022-12-01T10:43:19.3030330Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3030405Z Ran 1 test in 0.237s
2022-12-01T10:43:19.3030410Z 
2022-12-01T10:43:19.3030479Z OK (skipped=1)
2022-12-01T10:43:19.3030483Z 
2022-12-01T10:43:19.3030565Z Generating XML reports...
2022-12-01T10:43:19.3030891Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103828.xml
2022-12-01T10:43:19.3031124Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.3031397Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3031519Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3031796Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3031929Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3031934Z 
2022-12-01T10:43:19.3032004Z Running tests...
2022-12-01T10:43:19.3032195Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3032525Z   test_ddp_profiling_autograd_profiler (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'gloo', 'nccl'} (0.001s)
2022-12-01T10:43:19.3032530Z 
2022-12-01T10:43:19.3032753Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3032842Z Ran 1 test in 0.002s
2022-12-01T10:43:19.3032847Z 
2022-12-01T10:43:19.3032918Z OK (skipped=1)
2022-12-01T10:43:19.3032922Z 
2022-12-01T10:43:19.3033003Z Generating XML reports...
2022-12-01T10:43:19.3033338Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103830.xml
2022-12-01T10:43:19.3033571Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.3033842Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3033965Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3034243Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3034368Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3034385Z 
2022-12-01T10:43:19.3034444Z Running tests...
2022-12-01T10:43:19.3034637Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3034960Z   test_ddp_profiling_torch_profiler (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'gloo', 'nccl'} (0.002s)
2022-12-01T10:43:19.3034965Z 
2022-12-01T10:43:19.3035152Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3035226Z Ran 1 test in 0.002s
2022-12-01T10:43:19.3035230Z 
2022-12-01T10:43:19.3035299Z OK (skipped=1)
2022-12-01T10:43:19.3035304Z 
2022-12-01T10:43:19.3035385Z Generating XML reports...
2022-12-01T10:43:19.3035721Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103831.xml
2022-12-01T10:43:19.3035937Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.3036211Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3036336Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3036614Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3036749Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3036754Z 
2022-12-01T10:43:19.3036825Z Running tests...
2022-12-01T10:43:19.3037017Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3037251Z   test_ddp_python_error_logged (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2733
2022-12-01T10:43:19.3037404Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2734
2022-12-01T10:43:19.3037545Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2735
2022-12-01T10:43:19.3037816Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3037943Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3038220Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3038352Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3038529Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:43:19.3038797Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3038918Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3039182Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3039348Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3039592Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:43:19.3039864Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3039985Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3040259Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3040391Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3040565Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:43:19.3040861Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.3041148Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.3041440Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.3041602Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:43:19.3041774Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:43:19.3041938Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:43:19.3042046Z skip: Need at least 2 CUDA devices (1.856s)
2022-12-01T10:43:19.3042051Z 
2022-12-01T10:43:19.3042248Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3042310Z Ran 1 test in 1.856s
2022-12-01T10:43:19.3042315Z 
2022-12-01T10:43:19.3042387Z OK (skipped=1)
2022-12-01T10:43:19.3042391Z 
2022-12-01T10:43:19.3042476Z Generating XML reports...
2022-12-01T10:43:19.3042815Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103833.xml
2022-12-01T10:43:19.3043048Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.3043322Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3043445Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3043724Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3043846Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3043867Z 
2022-12-01T10:43:19.3043926Z Running tests...
2022-12-01T10:43:19.3044121Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3044829Z   test_ddp_returns_tensor_with_no_grad (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78595 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.234s)
2022-12-01T10:43:19.3044837Z 
2022-12-01T10:43:19.3045031Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3045107Z Ran 1 test in 0.234s
2022-12-01T10:43:19.3045112Z 
2022-12-01T10:43:19.3045183Z OK (skipped=1)
2022-12-01T10:43:19.3045187Z 
2022-12-01T10:43:19.3045271Z Generating XML reports...
2022-12-01T10:43:19.3045608Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103836.xml
2022-12-01T10:43:19.3045842Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.3046137Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3046292Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3046571Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3046704Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3046709Z 
2022-12-01T10:43:19.3046781Z Running tests...
2022-12-01T10:43:19.3046973Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3047302Z   test_ddp_shared_grad_acc_unused_params (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'nccl', 'gloo'} (0.002s)
2022-12-01T10:43:19.3047308Z 
2022-12-01T10:43:19.3047496Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3047572Z Ran 1 test in 0.002s
2022-12-01T10:43:19.3047577Z 
2022-12-01T10:43:19.3047634Z OK (skipped=1)
2022-12-01T10:43:19.3047653Z 
2022-12-01T10:43:19.3047724Z Generating XML reports...
2022-12-01T10:43:19.3048064Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103838.xml
2022-12-01T10:43:19.3048294Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.3048566Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3048690Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3048966Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3049099Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3049104Z 
2022-12-01T10:43:19.3049175Z Running tests...
2022-12-01T10:43:19.3049355Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3050053Z   test_ddp_static_graph_nested_types (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77625 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.244s)
2022-12-01T10:43:19.3050073Z 
2022-12-01T10:43:19.3050252Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3050330Z Ran 1 test in 0.244s
2022-12-01T10:43:19.3050335Z 
2022-12-01T10:43:19.3050405Z OK (skipped=1)
2022-12-01T10:43:19.3050409Z 
2022-12-01T10:43:19.3050492Z Generating XML reports...
2022-12-01T10:43:19.3050829Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103839.xml
2022-12-01T10:43:19.3051060Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.3051332Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3051458Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3051726Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3051861Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3051866Z 
2022-12-01T10:43:19.3051938Z Running tests...
2022-12-01T10:43:19.3052133Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3052370Z   test_ddp_sync_bn_training_vs_eval (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2816
2022-12-01T10:43:19.3052527Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2817
2022-12-01T10:43:19.3052712Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2818
2022-12-01T10:43:19.3052986Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3053127Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3053500Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3053641Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3053819Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:43:19.3054094Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3054220Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3054504Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3054640Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3054821Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:43:19.3055107Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.3055377Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3055502Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3055781Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3055915Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3056090Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:43:19.3056391Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.3056690Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.3056857Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:43:19.3057009Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:43:19.3057169Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:43:19.3057272Z skip: Need at least 2 CUDA devices (1.853s)
2022-12-01T10:43:19.3057278Z 
2022-12-01T10:43:19.3057472Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3057547Z Ran 1 test in 1.853s
2022-12-01T10:43:19.3057552Z 
2022-12-01T10:43:19.3057622Z OK (skipped=1)
2022-12-01T10:43:19.3057627Z 
2022-12-01T10:43:19.3057711Z Generating XML reports...
2022-12-01T10:43:19.3058048Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103841.xml
2022-12-01T10:43:19.3058282Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.3058541Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3058666Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3058946Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3059079Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3059084Z 
2022-12-01T10:43:19.3059155Z Running tests...
2022-12-01T10:43:19.3059348Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3059623Z   test_ddp_sync_module_states (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2869
2022-12-01T10:43:19.3059820Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2870
2022-12-01T10:43:19.3059959Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2871
2022-12-01T10:43:19.3060231Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3060476Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3060759Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3060894Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3061073Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:43:19.3061349Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3061475Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3061760Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3061882Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3062060Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:43:19.3062330Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3062453Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3062733Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3062871Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3063049Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:43:19.3063350Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.3063648Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.3063925Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.3064092Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:43:19.3064253Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:43:19.3064415Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:43:19.3064522Z skip: Need at least 2 CUDA devices (1.854s)
2022-12-01T10:43:19.3064527Z 
2022-12-01T10:43:19.3064725Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3064802Z Ran 1 test in 1.854s
2022-12-01T10:43:19.3064806Z 
2022-12-01T10:43:19.3064877Z OK (skipped=1)
2022-12-01T10:43:19.3064882Z 
2022-12-01T10:43:19.3065004Z Generating XML reports...
2022-12-01T10:43:19.3065345Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103844.xml
2022-12-01T10:43:19.3065576Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.3065848Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3065972Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3066249Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3066434Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3066473Z 
2022-12-01T10:43:19.3066549Z Running tests...
2022-12-01T10:43:19.3066745Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3066972Z   test_ddp_uneven_input_exception (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2922
2022-12-01T10:43:19.3067131Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2923
2022-12-01T10:43:19.3067284Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2924
2022-12-01T10:43:19.3067559Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3067683Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3067966Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3068103Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3068284Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:43:19.3068584Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.3068844Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3068968Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3069245Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3069377Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3069551Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:43:19.3069822Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3069947Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3070224Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3070346Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3070522Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:43:19.3070817Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.3071111Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.3071277Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:43:19.3071442Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:43:19.3071605Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:43:19.3071708Z skip: Need at least 2 CUDA devices (1.765s)
2022-12-01T10:43:19.3071713Z 
2022-12-01T10:43:19.3071908Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3071969Z Ran 1 test in 1.765s
2022-12-01T10:43:19.3071974Z 
2022-12-01T10:43:19.3072044Z OK (skipped=1)
2022-12-01T10:43:19.3072049Z 
2022-12-01T10:43:19.3072133Z Generating XML reports...
2022-12-01T10:43:19.3072473Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103847.xml
2022-12-01T10:43:19.3072703Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.3073012Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3073163Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3073441Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3073575Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3073580Z 
2022-12-01T10:43:19.3073640Z Running tests...
2022-12-01T10:43:19.3073835Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3074529Z   test_ddp_uneven_input_join_disable (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78684 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.237s)
2022-12-01T10:43:19.3074536Z 
2022-12-01T10:43:19.3074730Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3074806Z Ran 1 test in 0.238s
2022-12-01T10:43:19.3074811Z 
2022-12-01T10:43:19.3074881Z OK (skipped=1)
2022-12-01T10:43:19.3074886Z 
2022-12-01T10:43:19.3074969Z Generating XML reports...
2022-12-01T10:43:19.3075309Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103851.xml
2022-12-01T10:43:19.3075540Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.3075816Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3075927Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3076210Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3076347Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3076353Z 
2022-12-01T10:43:19.3076429Z Running tests...
2022-12-01T10:43:19.3076625Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3077302Z   test_ddp_uneven_inputs (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/75648 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.238s)
2022-12-01T10:43:19.3077308Z 
2022-12-01T10:43:19.3077500Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3077575Z Ran 1 test in 0.239s
2022-12-01T10:43:19.3077581Z 
2022-12-01T10:43:19.3077649Z OK (skipped=1)
2022-12-01T10:43:19.3077654Z 
2022-12-01T10:43:19.3077724Z Generating XML reports...
2022-12-01T10:43:19.3078061Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103852.xml
2022-12-01T10:43:19.3078293Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.3078565Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3078687Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3078967Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3079102Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3079107Z 
2022-12-01T10:43:19.3079179Z Running tests...
2022-12-01T10:43:19.3079371Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3080106Z   test_ddp_uneven_inputs_stop_iteration_sync_bn (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78113 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.238s)
2022-12-01T10:43:19.3080138Z 
2022-12-01T10:43:19.3080332Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3080394Z Ran 1 test in 0.238s
2022-12-01T10:43:19.3080399Z 
2022-12-01T10:43:19.3080469Z OK (skipped=1)
2022-12-01T10:43:19.3080474Z 
2022-12-01T10:43:19.3080556Z Generating XML reports...
2022-12-01T10:43:19.3080894Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103854.xml
2022-12-01T10:43:19.3081125Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.3081402Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3081526Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3081807Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3081928Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3081945Z 
2022-12-01T10:43:19.3082005Z Running tests...
2022-12-01T10:43:19.3082197Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3082543Z   test_ddp_unused_params_rebuild_buckets_exception (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'nccl', 'gloo'} (0.003s)
2022-12-01T10:43:19.3082548Z 
2022-12-01T10:43:19.3082740Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3082816Z Ran 1 test in 0.003s
2022-12-01T10:43:19.3082821Z 
2022-12-01T10:43:19.3082894Z OK (skipped=1)
2022-12-01T10:43:19.3082899Z 
2022-12-01T10:43:19.3082987Z Generating XML reports...
2022-12-01T10:43:19.3083325Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103855.xml
2022-12-01T10:43:19.3083550Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.3083823Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3083946Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3084224Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3084357Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3084362Z 
2022-12-01T10:43:19.3084433Z Running tests...
2022-12-01T10:43:19.3084625Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3084859Z   test_destroy_full_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3015
2022-12-01T10:43:19.3085016Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3016
2022-12-01T10:43:19.3085157Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3017
2022-12-01T10:43:19.3085430Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3085551Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3085829Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3085963Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3086140Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:43:19.3086439Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3086590Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3086853Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3086987Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3087163Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:43:19.3087431Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3087553Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3087830Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3087967Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3088143Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:43:19.3088439Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.3088723Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.3089014Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.3089178Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:43:19.3089340Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:43:19.3089502Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:43:19.3089679Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T10:43:19.3089854Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T10:43:19.3090026Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-12-01T10:43:19.3090316Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:43:19.3090595Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:43:19.3090885Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:43:19.3090953Z ok (2.053s)
2022-12-01T10:43:19.3090958Z 
2022-12-01T10:43:19.3091154Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3091228Z Ran 1 test in 2.053s
2022-12-01T10:43:19.3091235Z 
2022-12-01T10:43:19.3091295Z OK
2022-12-01T10:43:19.3091301Z 
2022-12-01T10:43:19.3091384Z Generating XML reports...
2022-12-01T10:43:19.3091722Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103857.xml
2022-12-01T10:43:19.3091954Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.3092218Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3092341Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3092619Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3092756Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3092761Z 
2022-12-01T10:43:19.3092833Z Running tests...
2022-12-01T10:43:19.3093063Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3093395Z   test_destroy_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3077
2022-12-01T10:43:19.3093552Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3078
2022-12-01T10:43:19.3093691Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3079
2022-12-01T10:43:19.3093967Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3094091Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3094373Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3094509Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3094694Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:43:19.3094967Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3095093Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3095372Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3095494Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3095673Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:43:19.3095943Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3096067Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3096347Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3096483Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3096659Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:43:19.3096959Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.3097260Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.3097537Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.3097703Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:43:19.3097870Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:43:19.3098048Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T10:43:19.3098225Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-12-01T10:43:19.3098388Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:43:19.3098556Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T10:43:19.3098848Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:43:19.3099142Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:43:19.3099417Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:43:19.3099486Z ok (2.052s)
2022-12-01T10:43:19.3099491Z 
2022-12-01T10:43:19.3099725Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3099805Z Ran 1 test in 2.052s
2022-12-01T10:43:19.3099837Z 
2022-12-01T10:43:19.3099899Z OK
2022-12-01T10:43:19.3099904Z 
2022-12-01T10:43:19.3099987Z Generating XML reports...
2022-12-01T10:43:19.3100328Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103900.xml
2022-12-01T10:43:19.3100661Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.3100920Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3101044Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3101329Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3101466Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3101472Z 
2022-12-01T10:43:19.3101547Z Running tests...
2022-12-01T10:43:19.3101742Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3102448Z   test_detect_ddp_is_actually_static (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78767 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.240s)
2022-12-01T10:43:19.3102454Z 
2022-12-01T10:43:19.3102650Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3102726Z Ran 1 test in 0.240s
2022-12-01T10:43:19.3102731Z 
2022-12-01T10:43:19.3102803Z OK (skipped=1)
2022-12-01T10:43:19.3102807Z 
2022-12-01T10:43:19.3102877Z Generating XML reports...
2022-12-01T10:43:19.3103220Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103904.xml
2022-12-01T10:43:19.3103452Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.3103730Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3103855Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3104139Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3104276Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3104280Z 
2022-12-01T10:43:19.3104351Z Running tests...
2022-12-01T10:43:19.3104547Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3104853Z   test_different_graph_across_ranks (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'gloo', 'nccl'} (0.002s)
2022-12-01T10:43:19.3104873Z 
2022-12-01T10:43:19.3105051Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3105128Z Ran 1 test in 0.002s
2022-12-01T10:43:19.3105133Z 
2022-12-01T10:43:19.3105204Z OK (skipped=1)
2022-12-01T10:43:19.3105209Z 
2022-12-01T10:43:19.3105292Z Generating XML reports...
2022-12-01T10:43:19.3105629Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103906.xml
2022-12-01T10:43:19.3105857Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.3106127Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3106248Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3106513Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3106702Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3106707Z 
2022-12-01T10:43:19.3106826Z Running tests...
2022-12-01T10:43:19.3107023Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3107264Z   test_dump_DDP_relevant_env_vars (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3156
2022-12-01T10:43:19.3107420Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3157
2022-12-01T10:43:19.3107574Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3158
2022-12-01T10:43:19.3107851Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3107961Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3108241Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3108379Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3108560Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:43:19.3108834Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3108958Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3109236Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3109372Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3109550Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:43:19.3109806Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3109930Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3110205Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3110342Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3110520Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:43:19.3110817Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.3111112Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.3111397Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.3111562Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:43:19.3111715Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:43:19.3111879Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:43:19.3111948Z ok (1.855s)
2022-12-01T10:43:19.3111954Z 
2022-12-01T10:43:19.3112149Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3112223Z Ran 1 test in 1.855s
2022-12-01T10:43:19.3112228Z 
2022-12-01T10:43:19.3112288Z OK
2022-12-01T10:43:19.3112292Z 
2022-12-01T10:43:19.3112375Z Generating XML reports...
2022-12-01T10:43:19.3112708Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103907.xml
2022-12-01T10:43:19.3112925Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.3113198Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3113356Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3113664Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3113798Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3113803Z 
2022-12-01T10:43:19.3113876Z Running tests...
2022-12-01T10:43:19.3114070Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3114288Z   test_gather (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3209
2022-12-01T10:43:19.3114442Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3210
2022-12-01T10:43:19.3114584Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3211
2022-12-01T10:43:19.3114853Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3114977Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3115258Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3115391Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3115569Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:43:19.3115841Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3115963Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3116242Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3116362Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3116541Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:43:19.3116813Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3116939Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3117215Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3117351Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3117528Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:43:19.3117827Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.3118110Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.3118403Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.3118569Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:43:19.3118816Z STAGE:2022-12-01 10:39:12 3210:3210 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.3118979Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:43:19.3119140Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:43:19.3119382Z STAGE:2022-12-01 10:39:12 3209:3209 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.3119621Z STAGE:2022-12-01 10:39:12 3211:3211 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.3119868Z STAGE:2022-12-01 10:39:12 3211:3211 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.3120132Z STAGE:2022-12-01 10:39:12 3210:3210 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.3120397Z STAGE:2022-12-01 10:39:12 3209:3209 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.3120634Z STAGE:2022-12-01 10:39:12 3211:3211 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.3120870Z STAGE:2022-12-01 10:39:12 3210:3210 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.3121114Z STAGE:2022-12-01 10:39:12 3211:3211 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.3121350Z STAGE:2022-12-01 10:39:12 3211:3211 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.3121585Z STAGE:2022-12-01 10:39:12 3209:3209 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.3121828Z STAGE:2022-12-01 10:39:12 3209:3209 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.3122069Z STAGE:2022-12-01 10:39:12 3210:3210 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.3122295Z STAGE:2022-12-01 10:39:12 3209:3209 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.3122536Z STAGE:2022-12-01 10:39:12 3209:3209 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.3122772Z STAGE:2022-12-01 10:39:12 3210:3210 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.3123015Z STAGE:2022-12-01 10:39:12 3210:3210 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.3123251Z STAGE:2022-12-01 10:39:12 3211:3211 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.3123319Z ok (2.054s)
2022-12-01T10:43:19.3123324Z 
2022-12-01T10:43:19.3123521Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3123598Z Ran 1 test in 2.054s
2022-12-01T10:43:19.3123603Z 
2022-12-01T10:43:19.3123650Z OK
2022-12-01T10:43:19.3123666Z 
2022-12-01T10:43:19.3123739Z Generating XML reports...
2022-12-01T10:43:19.3124080Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103910.xml
2022-12-01T10:43:19.3124311Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.3124583Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3124708Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3124985Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3125120Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3125126Z 
2022-12-01T10:43:19.3125197Z Running tests...
2022-12-01T10:43:19.3125375Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3125605Z   test_gather_checks (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3265
2022-12-01T10:43:19.3125762Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3266
2022-12-01T10:43:19.3125915Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3267
2022-12-01T10:43:19.3126188Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3126312Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3126592Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3126728Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3126892Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:43:19.3127199Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3127347Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3127627Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3127761Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3127937Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:43:19.3128206Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3128328Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3128602Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3128726Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3128901Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:43:19.3129198Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.3129495Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.3129787Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.3129952Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:43:19.3130115Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:43:19.3130276Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:43:19.3130343Z ok (1.855s)
2022-12-01T10:43:19.3130350Z 
2022-12-01T10:43:19.3130532Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3130610Z Ran 1 test in 1.855s
2022-12-01T10:43:19.3130614Z 
2022-12-01T10:43:19.3130675Z OK
2022-12-01T10:43:19.3130680Z 
2022-12-01T10:43:19.3130765Z Generating XML reports...
2022-12-01T10:43:19.3131105Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103914.xml
2022-12-01T10:43:19.3131338Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.3131612Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3131735Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3132000Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3132137Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3132144Z 
2022-12-01T10:43:19.3132215Z Running tests...
2022-12-01T10:43:19.3132410Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3132594Z   test_gather_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA gather (0.002s)
2022-12-01T10:43:19.3132600Z 
2022-12-01T10:43:19.3132791Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3132866Z Ran 1 test in 0.002s
2022-12-01T10:43:19.3132870Z 
2022-12-01T10:43:19.3132940Z OK (skipped=1)
2022-12-01T10:43:19.3132945Z 
2022-12-01T10:43:19.3133027Z Generating XML reports...
2022-12-01T10:43:19.3133429Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103917.xml
2022-12-01T10:43:19.3133671Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.3133985Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3134138Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3134416Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3134552Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3134557Z 
2022-12-01T10:43:19.3134630Z Running tests...
2022-12-01T10:43:19.3134822Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3135051Z   test_gather_full_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3328
2022-12-01T10:43:19.3135193Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3329
2022-12-01T10:43:19.3135346Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3330
2022-12-01T10:43:19.3135621Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3135747Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3136030Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3136164Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3136342Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:43:19.3136616Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3136725Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3137006Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3137144Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3137324Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:43:19.3137622Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.3137890Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3138014Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3138293Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3138427Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3138591Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:43:19.3138889Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.3139186Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.3139350Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:43:19.3139514Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:43:19.3139670Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:43:19.3139845Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T10:43:19.3140018Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T10:43:19.3140189Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-12-01T10:43:19.3140621Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:43:19.3140957Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:43:19.3141250Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:43:19.3141652Z STAGE:2022-12-01 10:39:20 3328:3328 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-12-01 10:39:20 3329:3329 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.3141657Z 
2022-12-01T10:43:19.3141901Z STAGE:2022-12-01 10:39:20 3330:3330 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.3142146Z STAGE:2022-12-01 10:39:20 3330:3330 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.3155260Z STAGE:2022-12-01 10:39:20 3329:3329 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.3155680Z STAGE:2022-12-01 10:39:20 3328:3328 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.3155962Z STAGE:2022-12-01 10:39:20 3330:3330 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.3156218Z STAGE:2022-12-01 10:39:20 3329:3329 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.3156466Z STAGE:2022-12-01 10:39:20 3330:3330 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.3156709Z STAGE:2022-12-01 10:39:20 3330:3330 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.3156937Z STAGE:2022-12-01 10:39:20 3328:3328 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.3157183Z STAGE:2022-12-01 10:39:20 3328:3328 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.3157430Z STAGE:2022-12-01 10:39:20 3329:3329 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.3157674Z STAGE:2022-12-01 10:39:20 3328:3328 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.3157923Z STAGE:2022-12-01 10:39:20 3328:3328 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.3158165Z STAGE:2022-12-01 10:39:20 3329:3329 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.3158408Z STAGE:2022-12-01 10:39:20 3329:3329 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.3158666Z STAGE:2022-12-01 10:39:20 3330:3330 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.3158724Z ok (1.955s)
2022-12-01T10:43:19.3158745Z 
2022-12-01T10:43:19.3158933Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3159013Z Ran 1 test in 1.955s
2022-12-01T10:43:19.3159018Z 
2022-12-01T10:43:19.3159096Z OK
2022-12-01T10:43:19.3159101Z 
2022-12-01T10:43:19.3159194Z Generating XML reports...
2022-12-01T10:43:19.3159541Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103918.xml
2022-12-01T10:43:19.3159782Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.3160096Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3160224Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3160496Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3160650Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3160655Z 
2022-12-01T10:43:19.3160731Z Running tests...
2022-12-01T10:43:19.3160932Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3161269Z   test_gather_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3393
2022-12-01T10:43:19.3161494Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3394
2022-12-01T10:43:19.3161652Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3395
2022-12-01T10:43:19.3161930Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3162056Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3162325Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3162465Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3162647Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:43:19.3162925Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3163054Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3163338Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3163475Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3163656Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:43:19.3163914Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3164040Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3164323Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3164462Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3164643Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:43:19.3164944Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.3165247Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.3165540Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.3165709Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:43:19.3165862Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:43:19.3166038Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-12-01T10:43:19.3166215Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T10:43:19.3166383Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:43:19.3166558Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T10:43:19.3166856Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:43:19.3167152Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:43:19.3167441Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:43:19.3167847Z STAGE:2022-12-01 10:39:23 3394:3394 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-12-01 10:39:23 3395:3395 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.3167853Z 
2022-12-01T10:43:19.3168133Z STAGE:2022-12-01 10:39:23 3395:3395 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.3168390Z STAGE:2022-12-01 10:39:23 3394:3394 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.3168634Z STAGE:2022-12-01 10:39:23 3395:3395 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.3168877Z STAGE:2022-12-01 10:39:23 3394:3394 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.3169127Z STAGE:2022-12-01 10:39:23 3394:3394 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.3169371Z STAGE:2022-12-01 10:39:23 3395:3395 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.3169442Z ok (2.055s)
2022-12-01T10:43:19.3169447Z 
2022-12-01T10:43:19.3169646Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3169725Z Ran 1 test in 2.055s
2022-12-01T10:43:19.3169730Z 
2022-12-01T10:43:19.3169782Z OK
2022-12-01T10:43:19.3169800Z 
2022-12-01T10:43:19.3169875Z Generating XML reports...
2022-12-01T10:43:19.3170221Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103922.xml
2022-12-01T10:43:19.3170460Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.3170737Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3170864Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3171147Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3171287Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3171292Z 
2022-12-01T10:43:19.3171365Z Running tests...
2022-12-01T10:43:19.3171552Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3171786Z   test_gather_object (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3454
2022-12-01T10:43:19.3171947Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3455
2022-12-01T10:43:19.3172102Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3456
2022-12-01T10:43:19.3172377Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3172503Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3172782Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3172920Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3173099Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:43:19.3173444Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3173575Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3173859Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3173997Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3174178Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:43:19.3174449Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3174579Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3174858Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3175016Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3175221Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:43:19.3175523Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.3175841Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.3176134Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.3176302Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:43:19.3176462Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:43:19.3176627Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:43:19.3176700Z ok (1.854s)
2022-12-01T10:43:19.3176722Z 
2022-12-01T10:43:19.3176909Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3176988Z Ran 1 test in 1.855s
2022-12-01T10:43:19.3176994Z 
2022-12-01T10:43:19.3177057Z OK
2022-12-01T10:43:19.3177062Z 
2022-12-01T10:43:19.3177149Z Generating XML reports...
2022-12-01T10:43:19.3177507Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103925.xml
2022-12-01T10:43:19.3177745Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.3178024Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3178151Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3178436Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3178562Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3178569Z 
2022-12-01T10:43:19.3178644Z Running tests...
2022-12-01T10:43:19.3178873Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3186152Z   test_gather_object_subgroup (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/82866 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.242s)
2022-12-01T10:43:19.3186169Z 
2022-12-01T10:43:19.3186483Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3186590Z Ran 1 test in 0.242s
2022-12-01T10:43:19.3186597Z 
2022-12-01T10:43:19.3186704Z OK (skipped=1)
2022-12-01T10:43:19.3186710Z 
2022-12-01T10:43:19.3186840Z Generating XML reports...
2022-12-01T10:43:19.3187341Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103929.xml
2022-12-01T10:43:19.3187713Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.3188149Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3188335Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3188792Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3189002Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3189009Z 
2022-12-01T10:43:19.3189112Z Running tests...
2022-12-01T10:43:19.3189414Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3189839Z   test_get_backend (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3517
2022-12-01T10:43:19.3190073Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3518
2022-12-01T10:43:19.3190290Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3519
2022-12-01T10:43:19.3190738Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3190920Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3191396Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3191616Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3191911Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:43:19.3192376Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3192585Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3193070Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3193293Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3193565Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:43:19.3194017Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3194205Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3194680Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3194892Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3195184Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:43:19.3195692Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.3196199Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.3196702Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.3196969Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:43:19.3197228Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:43:19.3197491Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:43:19.3197785Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T10:43:19.3198074Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-12-01T10:43:19.3198352Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T10:43:19.3219467Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:43:19.3219866Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:43:19.3220154Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:43:19.3220216Z ok (1.953s)
2022-12-01T10:43:19.3220224Z 
2022-12-01T10:43:19.3220596Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3220668Z Ran 1 test in 1.953s
2022-12-01T10:43:19.3220674Z 
2022-12-01T10:43:19.3220728Z OK
2022-12-01T10:43:19.3220857Z 
2022-12-01T10:43:19.3220936Z Generating XML reports...
2022-12-01T10:43:19.3221330Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103930.xml
2022-12-01T10:43:19.3221558Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.3221827Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3221945Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3222214Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3222342Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3222348Z 
2022-12-01T10:43:19.3222414Z Running tests...
2022-12-01T10:43:19.3222602Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3222823Z   test_get_future (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3576
2022-12-01T10:43:19.3222974Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3577
2022-12-01T10:43:19.3223117Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3578
2022-12-01T10:43:19.3223388Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3223504Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3223773Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3223901Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3224071Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:43:19.3224338Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3224455Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3224728Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3224856Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3225029Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:43:19.3225315Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.3225577Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3225695Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3225973Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3226102Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3226271Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:43:19.3226562Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.3226852Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.3227010Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:43:19.3227162Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:43:19.3227319Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:43:19.3227414Z ok (1.762s)
2022-12-01T10:43:19.3227422Z 
2022-12-01T10:43:19.3227640Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3227708Z Ran 1 test in 1.762s
2022-12-01T10:43:19.3227714Z 
2022-12-01T10:43:19.3227768Z OK
2022-12-01T10:43:19.3227772Z 
2022-12-01T10:43:19.3227849Z Generating XML reports...
2022-12-01T10:43:19.3228184Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103934.xml
2022-12-01T10:43:19.3228411Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.3228672Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3228789Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3229062Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3229196Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3229203Z 
2022-12-01T10:43:19.3229268Z Running tests...
2022-12-01T10:43:19.3229457Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3229673Z   test_get_rank (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3629
2022-12-01T10:43:19.3229828Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3630
2022-12-01T10:43:19.3229966Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3631
2022-12-01T10:43:19.3230234Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3230349Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3230624Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3230755Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3230932Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:43:19.3231196Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3231312Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3231584Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3231704Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3231876Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:43:19.3232139Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3232257Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3232531Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3232656Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3232824Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:43:19.3233116Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.3233408Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.3233684Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.3233844Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:43:19.3234026Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:43:19.3234208Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:43:19.3234270Z ok (1.954s)
2022-12-01T10:43:19.3234275Z 
2022-12-01T10:43:19.3234466Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3234537Z Ran 1 test in 1.955s
2022-12-01T10:43:19.3234542Z 
2022-12-01T10:43:19.3234596Z OK
2022-12-01T10:43:19.3234601Z 
2022-12-01T10:43:19.3234671Z Generating XML reports...
2022-12-01T10:43:19.3235007Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103937.xml
2022-12-01T10:43:19.3235236Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.3235506Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3235626Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3235903Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3236032Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3236037Z 
2022-12-01T10:43:19.3236105Z Running tests...
2022-12-01T10:43:19.3236292Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3236516Z   test_get_rank_size_full_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3682
2022-12-01T10:43:19.3236665Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3683
2022-12-01T10:43:19.3236810Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3684
2022-12-01T10:43:19.3237078Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3237196Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3237471Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3237607Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3237784Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:43:19.3238054Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3238176Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3238455Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3238577Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3238754Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:43:19.3239024Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3239147Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3239428Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3239562Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3239736Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:43:19.3240033Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.3240316Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.3240641Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.3240850Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:43:19.3241014Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:43:19.3241174Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:43:19.3241348Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T10:43:19.3241521Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T10:43:19.3241813Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:43:19.3241983Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-12-01T10:43:19.3242265Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:43:19.3242557Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:43:19.3242625Z ok (1.954s)
2022-12-01T10:43:19.3242630Z 
2022-12-01T10:43:19.3242825Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3242900Z Ran 1 test in 1.955s
2022-12-01T10:43:19.3242905Z 
2022-12-01T10:43:19.3242968Z OK
2022-12-01T10:43:19.3242973Z 
2022-12-01T10:43:19.3243059Z Generating XML reports...
2022-12-01T10:43:19.3243401Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103940.xml
2022-12-01T10:43:19.3243638Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.3243902Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3244030Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3244310Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3244446Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3244451Z 
2022-12-01T10:43:19.3244526Z Running tests...
2022-12-01T10:43:19.3244719Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3244953Z   test_get_rank_size_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3744
2022-12-01T10:43:19.3245110Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3745
2022-12-01T10:43:19.3245261Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3746
2022-12-01T10:43:19.3245523Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3245646Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3245923Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3246055Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3246229Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:43:19.3246502Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3246624Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3246904Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3247054Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3247256Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:43:19.3247529Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3247652Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3247931Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3248063Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3248238Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:43:19.3248535Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.3248835Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.3249121Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.3249285Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:43:19.3249448Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:43:19.3249609Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:43:19.3249782Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T10:43:19.3249955Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T10:43:19.3250127Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-12-01T10:43:19.3250425Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:43:19.3250721Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:43:19.3251001Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:43:19.3251069Z ok (1.955s)
2022-12-01T10:43:19.3251074Z 
2022-12-01T10:43:19.3251271Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3251348Z Ran 1 test in 1.955s
2022-12-01T10:43:19.3251353Z 
2022-12-01T10:43:19.3251415Z OK
2022-12-01T10:43:19.3251420Z 
2022-12-01T10:43:19.3251503Z Generating XML reports...
2022-12-01T10:43:19.3251846Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103944.xml
2022-12-01T10:43:19.3252081Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.3252356Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3252469Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3252749Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3252886Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3252892Z 
2022-12-01T10:43:19.3252963Z Running tests...
2022-12-01T10:43:19.3253155Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3253547Z   test_invalid_static_graph (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'gloo', 'nccl'} (0.002s)
2022-12-01T10:43:19.3253553Z 
2022-12-01T10:43:19.3253747Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3253854Z Ran 1 test in 0.003s
2022-12-01T10:43:19.3253860Z 
2022-12-01T10:43:19.3253957Z OK (skipped=1)
2022-12-01T10:43:19.3253962Z 
2022-12-01T10:43:19.3254033Z Generating XML reports...
2022-12-01T10:43:19.3254374Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103947.xml
2022-12-01T10:43:19.3254612Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.3254884Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3255006Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3255285Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3255416Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3255421Z 
2022-12-01T10:43:19.3255492Z Running tests...
2022-12-01T10:43:19.3255676Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3255897Z   test_irecv (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3813
2022-12-01T10:43:19.3256050Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3814
2022-12-01T10:43:19.3256206Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3815
2022-12-01T10:43:19.3256480Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3256605Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3256882Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3257016Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3257195Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:43:19.3257455Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3257577Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3257857Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3257993Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3258170Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:43:19.3258443Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3258564Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3258842Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3258963Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3259136Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:43:19.3259433Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.3259731Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.3260023Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.3260186Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:43:19.3260461Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:43:19.3260701Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:43:19.3260805Z ok (1.958s)
2022-12-01T10:43:19.3260811Z 
2022-12-01T10:43:19.3260997Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3261072Z Ran 1 test in 1.958s
2022-12-01T10:43:19.3261077Z 
2022-12-01T10:43:19.3261138Z OK
2022-12-01T10:43:19.3261142Z 
2022-12-01T10:43:19.3261227Z Generating XML reports...
2022-12-01T10:43:19.3261569Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103949.xml
2022-12-01T10:43:19.3261799Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.3262072Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3262196Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3262479Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3262604Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3262608Z 
2022-12-01T10:43:19.3262681Z Running tests...
2022-12-01T10:43:19.3262877Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3263097Z   test_isend (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3866
2022-12-01T10:43:19.3263255Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3867
2022-12-01T10:43:19.3263410Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3868
2022-12-01T10:43:19.3263682Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3263805Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3264074Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3264209Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3264386Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:43:19.3264655Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3264778Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3265054Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3265187Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3265363Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:43:19.3265665Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.3265921Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3266042Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3266316Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3266450Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3266624Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:43:19.3266918Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.3267212Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.3267409Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:43:19.3267595Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:43:19.3267744Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:43:19.3267810Z ok (1.854s)
2022-12-01T10:43:19.3267815Z 
2022-12-01T10:43:19.3268009Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3268087Z Ran 1 test in 1.854s
2022-12-01T10:43:19.3268092Z 
2022-12-01T10:43:19.3268153Z OK
2022-12-01T10:43:19.3268157Z 
2022-12-01T10:43:19.3268241Z Generating XML reports...
2022-12-01T10:43:19.3268583Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103952.xml
2022-12-01T10:43:19.3268814Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.3269077Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3269204Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3269484Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3269618Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3269622Z 
2022-12-01T10:43:19.3269694Z Running tests...
2022-12-01T10:43:19.3269887Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3270126Z   test_isend_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3919
2022-12-01T10:43:19.3270280Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3920
2022-12-01T10:43:19.3270432Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3921
2022-12-01T10:43:19.3270694Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3270820Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3271100Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3271233Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3271410Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:43:19.3271679Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3271806Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3272085Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3272210Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3272388Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:43:19.3272657Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3272781Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3273059Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3273192Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3273367Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:43:19.3273663Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.3274023Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.3274332Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.3274498Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:43:19.3274661Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:43:19.3274825Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:43:19.3275070Z STAGE:2022-12-01 10:39:57 3921:3921 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.3275317Z STAGE:2022-12-01 10:39:57 3919:3919 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.3275562Z STAGE:2022-12-01 10:39:57 3920:3920 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.3275813Z STAGE:2022-12-01 10:39:57 3919:3919 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.3276059Z STAGE:2022-12-01 10:39:57 3921:3921 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.3276290Z STAGE:2022-12-01 10:39:57 3920:3920 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.3276360Z ok (1.856s)
2022-12-01T10:43:19.3276365Z 
2022-12-01T10:43:19.3276561Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3276638Z Ran 1 test in 1.856s
2022-12-01T10:43:19.3276643Z 
2022-12-01T10:43:19.3276703Z OK
2022-12-01T10:43:19.3276708Z 
2022-12-01T10:43:19.3276793Z Generating XML reports...
2022-12-01T10:43:19.3277134Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103955.xml
2022-12-01T10:43:19.3277369Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.3277632Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3277757Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3278039Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3278175Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3278180Z 
2022-12-01T10:43:19.3278252Z Running tests...
2022-12-01T10:43:19.3278449Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3278681Z   test_isend_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3975
2022-12-01T10:43:19.3278836Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3976
2022-12-01T10:43:19.3278986Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3977
2022-12-01T10:43:19.3279281Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3279405Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3279683Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3279817Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3279997Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:43:19.3280270Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3280394Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3280674Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3280836Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3281026Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:43:19.3281296Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3281418Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3281697Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3281830Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3282009Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:43:19.3282308Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.3282605Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.3282903Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.3283056Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:43:19.3283299Z STAGE:2022-12-01 10:40:00 3976:3976 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.3283462Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:43:19.3283626Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:43:19.3283865Z STAGE:2022-12-01 10:40:00 3975:3975 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.3284105Z STAGE:2022-12-01 10:40:00 3977:3977 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.3284353Z STAGE:2022-12-01 10:40:00 3975:3975 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.3284594Z STAGE:2022-12-01 10:40:00 3977:3977 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.3284818Z STAGE:2022-12-01 10:40:00 3976:3976 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.3284884Z ok (1.955s)
2022-12-01T10:43:19.3284889Z 
2022-12-01T10:43:19.3285085Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3285160Z Ran 1 test in 1.955s
2022-12-01T10:43:19.3285165Z 
2022-12-01T10:43:19.3285227Z OK
2022-12-01T10:43:19.3285232Z 
2022-12-01T10:43:19.3285315Z Generating XML reports...
2022-12-01T10:43:19.3285661Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103959.xml
2022-12-01T10:43:19.3285896Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.3286173Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3286289Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3286570Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3286705Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3286710Z 
2022-12-01T10:43:19.3286789Z Running tests...
2022-12-01T10:43:19.3286982Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3287316Z   test_monitored_barrier_allreduce_hang (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'gloo', 'nccl'} (0.002s)
2022-12-01T10:43:19.3287321Z 
2022-12-01T10:43:19.3287511Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3287587Z Ran 1 test in 0.002s
2022-12-01T10:43:19.3287591Z 
2022-12-01T10:43:19.3287691Z OK (skipped=1)
2022-12-01T10:43:19.3287734Z 
2022-12-01T10:43:19.3287806Z Generating XML reports...
2022-12-01T10:43:19.3288148Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104002.xml
2022-12-01T10:43:19.3288380Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.3288653Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3288775Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3289053Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3289187Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3289191Z 
2022-12-01T10:43:19.3289262Z Running tests...
2022-12-01T10:43:19.3289443Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3289802Z   test_monitored_barrier_allreduce_hang_wait_all_ranks (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'gloo', 'nccl'} (0.001s)
2022-12-01T10:43:19.3289807Z 
2022-12-01T10:43:19.3289995Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3290071Z Ran 1 test in 0.002s
2022-12-01T10:43:19.3290076Z 
2022-12-01T10:43:19.3290146Z OK (skipped=1)
2022-12-01T10:43:19.3290150Z 
2022-12-01T10:43:19.3290233Z Generating XML reports...
2022-12-01T10:43:19.3290573Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104003.xml
2022-12-01T10:43:19.3290806Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.3291080Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3291192Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3291474Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3291608Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3291614Z 
2022-12-01T10:43:19.3291685Z Running tests...
2022-12-01T10:43:19.3291880Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3292124Z   test_monitored_barrier_failure_order (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4051
2022-12-01T10:43:19.3292279Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4052
2022-12-01T10:43:19.3292432Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4053
2022-12-01T10:43:19.3292709Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3292820Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3293101Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3293243Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3293501Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:43:19.3293775Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3293899Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3294183Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3294318Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3294518Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:43:19.3294813Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3294936Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3295210Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3295343Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3295518Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:43:19.3295817Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.3296114Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.3296407Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.3296574Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:43:19.3296725Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:43:19.3296890Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:43:19.3297149Z [E ProcessGroupGloo.cpp:137] Rank 1 successfully reached monitoredBarrier, but received errors while waiting for send/recv from rank 0. Please check rank 0 logs for faulty rank.
2022-12-01T10:43:19.3297315Z [E ProcessGroupGloo.cpp:137] [Rank 0]: Rank 2 failed to pass monitoredBarrier in 2000 ms
2022-12-01T10:43:19.3297383Z ok (3.856s)
2022-12-01T10:43:19.3297390Z 
2022-12-01T10:43:19.3297585Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3297663Z Ran 1 test in 3.857s
2022-12-01T10:43:19.3297668Z 
2022-12-01T10:43:19.3297730Z OK
2022-12-01T10:43:19.3297734Z 
2022-12-01T10:43:19.3297805Z Generating XML reports...
2022-12-01T10:43:19.3298146Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104005.xml
2022-12-01T10:43:19.3298375Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.3298647Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3298771Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3299052Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3299187Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3299192Z 
2022-12-01T10:43:19.3299264Z Running tests...
2022-12-01T10:43:19.3299461Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3299687Z   test_monitored_barrier_gloo (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4104
2022-12-01T10:43:19.3299840Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4105
2022-12-01T10:43:19.3299992Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4106
2022-12-01T10:43:19.3300264Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3300547Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3300835Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3300970Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3301208Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:43:19.3301501Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3301625Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3301907Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3302043Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3302225Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:43:19.3302498Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3302620Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3302902Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3303037Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3303203Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:43:19.3303498Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.3303794Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.3304089Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.3304253Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:43:19.3304414Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:43:19.3304579Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:43:19.3304747Z [E ProcessGroupGloo.cpp:137] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 2000 ms
2022-12-01T10:43:19.3305004Z [E ProcessGroupGloo.cpp:137] Rank 2 successfully reached monitoredBarrier, but received errors while waiting for send/recv from rank 0. Please check rank 0 logs for faulty rank.
2022-12-01T10:43:19.3305059Z ok (3.958s)
2022-12-01T10:43:19.3305075Z 
2022-12-01T10:43:19.3305258Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3305333Z Ran 1 test in 3.958s
2022-12-01T10:43:19.3305338Z 
2022-12-01T10:43:19.3305399Z OK
2022-12-01T10:43:19.3305404Z 
2022-12-01T10:43:19.3305486Z Generating XML reports...
2022-12-01T10:43:19.3305824Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104010.xml
2022-12-01T10:43:19.3306056Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.3306330Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3306457Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3306725Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3306859Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3306864Z 
2022-12-01T10:43:19.3306933Z Running tests...
2022-12-01T10:43:19.3307128Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3307374Z   test_monitored_barrier_gloo_rank_0_timeout (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4157
2022-12-01T10:43:19.3307532Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4158
2022-12-01T10:43:19.3307712Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4159
2022-12-01T10:43:19.3308014Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3308123Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3308400Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3308533Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3308709Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:43:19.3308982Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3309104Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3309387Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3309521Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3309697Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:43:19.3309955Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3310079Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3310353Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3310486Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3310661Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:43:19.3310964Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.3311260Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.3311557Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.3311721Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:43:19.3311872Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:43:19.3312034Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:43:19.3312209Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T10:43:19.3312384Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-12-01T10:43:19.3312555Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T10:43:19.3312851Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:43:19.3313148Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:43:19.3313438Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:43:19.3313594Z [E ProcessGroupGloo.cpp:137] Rank 0 timed out in monitoredBarrier after 0 ms.
2022-12-01T10:43:19.3313704Z No ranks successfully processed in monitoredBarrier.
2022-12-01T10:43:19.3313871Z [E ProcessGroupGloo.cpp:137] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 0 ms
2022-12-01T10:43:19.3313939Z ok (1.855s)
2022-12-01T10:43:19.3313944Z 
2022-12-01T10:43:19.3314141Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3314216Z Ran 1 test in 1.855s
2022-12-01T10:43:19.3314252Z 
2022-12-01T10:43:19.3314314Z OK
2022-12-01T10:43:19.3314347Z 
2022-12-01T10:43:19.3314433Z Generating XML reports...
2022-12-01T10:43:19.3314774Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104016.xml
2022-12-01T10:43:19.3314990Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.3315262Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3315386Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3315667Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3315800Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3315805Z 
2022-12-01T10:43:19.3315877Z Running tests...
2022-12-01T10:43:19.3316074Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3316321Z   test_monitored_barrier_gloo_subgroup (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4219
2022-12-01T10:43:19.3316475Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4220
2022-12-01T10:43:19.3316615Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4221
2022-12-01T10:43:19.3316886Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3317009Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3317287Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3317422Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3317599Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:43:19.3317873Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3317997Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3318265Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3318400Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3318576Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:43:19.3318872Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.3319138Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3319261Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3319542Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3319679Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3319853Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:43:19.3320136Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.3320430Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.3320594Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:43:19.3320758Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:43:19.3320961Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-12-01T10:43:19.3321147Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:43:19.3321316Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T10:43:19.3321490Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T10:43:19.3321781Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:43:19.3322058Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:43:19.3322346Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:43:19.3322512Z [E ProcessGroupGloo.cpp:137] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 100 ms
2022-12-01T10:43:19.3322883Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:370: UserWarning: Running monitored_barrier on global rank 2 which does not belong to the given group.
2022-12-01T10:43:19.3323017Z   f"Running {op_name} on global rank {global_rank} which does not "
2022-12-01T10:43:19.3323084Z ok (2.057s)
2022-12-01T10:43:19.3323090Z 
2022-12-01T10:43:19.3323286Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3323361Z Ran 1 test in 2.057s
2022-12-01T10:43:19.3323366Z 
2022-12-01T10:43:19.3323426Z OK
2022-12-01T10:43:19.3323430Z 
2022-12-01T10:43:19.3323500Z Generating XML reports...
2022-12-01T10:43:19.3323838Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104019.xml
2022-12-01T10:43:19.3324071Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.3324349Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3324474Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3324753Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3324889Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3324894Z 
2022-12-01T10:43:19.3324966Z Running tests...
2022-12-01T10:43:19.3325159Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3325389Z   test_monitored_barrier_wait_all_ranks (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4278
2022-12-01T10:43:19.3325542Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4279
2022-12-01T10:43:19.3325695Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4280
2022-12-01T10:43:19.3325969Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3326093Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3326370Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3326505Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3326684Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:43:19.3326940Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3327066Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3327345Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3327515Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3327721Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:43:19.3327993Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3328118Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3328392Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3328524Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3328686Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:43:19.3328984Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.3329281Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.3329572Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.3329738Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:43:19.3329902Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:43:19.3330064Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:43:19.3330226Z [E ProcessGroupGloo.cpp:2803] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 100 ms
2022-12-01T10:43:19.3330387Z [E ProcessGroupGloo.cpp:2803] [Rank 0]: Rank 2 failed to pass monitoredBarrier in 100 ms
2022-12-01T10:43:19.3330536Z [E ProcessGroupGloo.cpp:137] [Rank 0]: Ranks 1, 2 failed to pass monitoredBarrier in 100 ms
2022-12-01T10:43:19.3330603Z ok (2.054s)
2022-12-01T10:43:19.3330608Z 
2022-12-01T10:43:19.3330806Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3330885Z Ran 1 test in 2.054s
2022-12-01T10:43:19.3330890Z 
2022-12-01T10:43:19.3330950Z OK
2022-12-01T10:43:19.3330955Z 
2022-12-01T10:43:19.3331038Z Generating XML reports...
2022-12-01T10:43:19.3331378Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104022.xml
2022-12-01T10:43:19.3331611Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.3331869Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3331992Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3332267Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3332403Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3332408Z 
2022-12-01T10:43:19.3332484Z Running tests...
2022-12-01T10:43:19.3332679Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3332981Z   test_nccl_backend_bool_allgather (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl'} (0.002s)
2022-12-01T10:43:19.3332987Z 
2022-12-01T10:43:19.3333182Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3333337Z Ran 1 test in 0.002s
2022-12-01T10:43:19.3333344Z 
2022-12-01T10:43:19.3333403Z OK (skipped=1)
2022-12-01T10:43:19.3333422Z 
2022-12-01T10:43:19.3333493Z Generating XML reports...
2022-12-01T10:43:19.3333832Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104026.xml
2022-12-01T10:43:19.3334065Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.3334385Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3334551Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3334833Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3334965Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3334970Z 
2022-12-01T10:43:19.3335044Z Running tests...
2022-12-01T10:43:19.3335226Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3335526Z   test_nccl_backend_bool_allreduce (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl'} (0.002s)
2022-12-01T10:43:19.3335531Z 
2022-12-01T10:43:19.3335721Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3335794Z Ran 1 test in 0.002s
2022-12-01T10:43:19.3335798Z 
2022-12-01T10:43:19.3335870Z OK (skipped=1)
2022-12-01T10:43:19.3335875Z 
2022-12-01T10:43:19.3335960Z Generating XML reports...
2022-12-01T10:43:19.3336301Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104027.xml
2022-12-01T10:43:19.3336535Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.3336810Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3336920Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3337200Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3337337Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3337342Z 
2022-12-01T10:43:19.3337412Z Running tests...
2022-12-01T10:43:19.3337609Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3337915Z   test_nccl_backend_bool_broadcast (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl'} (0.002s)
2022-12-01T10:43:19.3337923Z 
2022-12-01T10:43:19.3338115Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3338190Z Ran 1 test in 0.002s
2022-12-01T10:43:19.3338195Z 
2022-12-01T10:43:19.3338254Z OK (skipped=1)
2022-12-01T10:43:19.3338270Z 
2022-12-01T10:43:19.3338339Z Generating XML reports...
2022-12-01T10:43:19.3338671Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104029.xml
2022-12-01T10:43:19.3338902Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.3339173Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3339293Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3339570Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3339705Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3339710Z 
2022-12-01T10:43:19.3339781Z Running tests...
2022-12-01T10:43:19.3339960Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3340263Z   test_nccl_backend_bool_reduce (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl'} (0.002s)
2022-12-01T10:43:19.3340268Z 
2022-12-01T10:43:19.3340603Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3340680Z Ran 1 test in 0.003s
2022-12-01T10:43:19.3340685Z 
2022-12-01T10:43:19.3340757Z OK (skipped=1)
2022-12-01T10:43:19.3340761Z 
2022-12-01T10:43:19.3340845Z Generating XML reports...
2022-12-01T10:43:19.3341249Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104030.xml
2022-12-01T10:43:19.3341521Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.3341796Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3341907Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3342189Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3342326Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3342331Z 
2022-12-01T10:43:19.3342404Z Running tests...
2022-12-01T10:43:19.3342601Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3342820Z   test_nccl_high_priority_stream (__main__.TestDistBackendWithSpawn) ... skip: Only NCCL backend supports high priority stream (0.002s)
2022-12-01T10:43:19.3342827Z 
2022-12-01T10:43:19.3343022Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3343099Z Ran 1 test in 0.002s
2022-12-01T10:43:19.3343103Z 
2022-12-01T10:43:19.3343177Z OK (skipped=1)
2022-12-01T10:43:19.3343182Z 
2022-12-01T10:43:19.3343252Z Generating XML reports...
2022-12-01T10:43:19.3343591Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104031.xml
2022-12-01T10:43:19.3343822Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.3344096Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3344220Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3344500Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3344638Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3344645Z 
2022-12-01T10:43:19.3344719Z Running tests...
2022-12-01T10:43:19.3344900Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3345084Z   test_new_subgroups (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2022-12-01T10:43:19.3345089Z 
2022-12-01T10:43:19.3345278Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3345352Z Ran 1 test in 0.002s
2022-12-01T10:43:19.3345356Z 
2022-12-01T10:43:19.3345426Z OK (skipped=1)
2022-12-01T10:43:19.3345430Z 
2022-12-01T10:43:19.3345512Z Generating XML reports...
2022-12-01T10:43:19.3345846Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104033.xml
2022-12-01T10:43:19.3346078Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.3346354Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3346465Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3346742Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3346873Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3346878Z 
2022-12-01T10:43:19.3346949Z Running tests...
2022-12-01T10:43:19.3347138Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3347332Z   test_new_subgroups_by_enumeration (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2022-12-01T10:43:19.3347337Z 
2022-12-01T10:43:19.3347527Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3347600Z Ran 1 test in 0.002s
2022-12-01T10:43:19.3347605Z 
2022-12-01T10:43:19.3347674Z OK (skipped=1)
2022-12-01T10:43:19.3347710Z 
2022-12-01T10:43:19.3347781Z Generating XML reports...
2022-12-01T10:43:19.3348157Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104034.xml
2022-12-01T10:43:19.3348386Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.3348657Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3348781Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3349058Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3349190Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3349195Z 
2022-12-01T10:43:19.3349299Z Running tests...
2022-12-01T10:43:19.3349476Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3349704Z   test_new_subgroups_by_enumeration_input_rank_exceeds_world_size (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2022-12-01T10:43:19.3349712Z 
2022-12-01T10:43:19.3349900Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3349972Z Ran 1 test in 0.002s
2022-12-01T10:43:19.3349977Z 
2022-12-01T10:43:19.3350047Z OK (skipped=1)
2022-12-01T10:43:19.3350051Z 
2022-12-01T10:43:19.3350133Z Generating XML reports...
2022-12-01T10:43:19.3350464Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104035.xml
2022-12-01T10:43:19.3350697Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.3350965Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3351077Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3351353Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3351490Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3351494Z 
2022-12-01T10:43:19.3351565Z Running tests...
2022-12-01T10:43:19.3351758Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3352020Z   test_new_subgroups_by_enumeration_negative_input_rank (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4411
2022-12-01T10:43:19.3352176Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4412
2022-12-01T10:43:19.3352329Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4413
2022-12-01T10:43:19.3352598Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3352710Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3352991Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3353126Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3353303Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:43:19.3353570Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3353692Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3353972Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3354104Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3354311Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:43:19.3354625Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.3354895Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3355017Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3355292Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3355425Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3355600Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:43:19.3355902Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.3356201Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.3356369Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:43:19.3356519Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:43:19.3356681Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:43:19.3356776Z skip: CUDA is not available. (1.764s)
2022-12-01T10:43:19.3356782Z 
2022-12-01T10:43:19.3356977Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3357051Z Ran 1 test in 1.765s
2022-12-01T10:43:19.3357056Z 
2022-12-01T10:43:19.3357128Z OK (skipped=1)
2022-12-01T10:43:19.3357132Z 
2022-12-01T10:43:19.3357217Z Generating XML reports...
2022-12-01T10:43:19.3357558Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104037.xml
2022-12-01T10:43:19.3357780Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.3358057Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3358179Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3358457Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3358588Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3358593Z 
2022-12-01T10:43:19.3358664Z Running tests...
2022-12-01T10:43:19.3358857Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3359112Z   test_new_subgroups_group_size_exceeds_world_size (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4464
2022-12-01T10:43:19.3359270Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4465
2022-12-01T10:43:19.3359410Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4466
2022-12-01T10:43:19.3359683Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3359805Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3360085Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3360218Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3360394Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:43:19.3360665Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3360787Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3361098Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3361245Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3361420Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:43:19.3361687Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3361809Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3362090Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3362222Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3362395Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:43:19.3362693Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.3362981Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.3363270Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.3363433Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:43:19.3363594Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:43:19.3363755Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:43:19.3363850Z skip: CUDA is not available. (1.754s)
2022-12-01T10:43:19.3363855Z 
2022-12-01T10:43:19.3364049Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3364125Z Ran 1 test in 1.755s
2022-12-01T10:43:19.3364130Z 
2022-12-01T10:43:19.3364199Z OK (skipped=1)
2022-12-01T10:43:19.3364205Z 
2022-12-01T10:43:19.3364276Z Generating XML reports...
2022-12-01T10:43:19.3364616Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104040.xml
2022-12-01T10:43:19.3364847Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.3365119Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3365241Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3365520Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3365654Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3365659Z 
2022-12-01T10:43:19.3365729Z Running tests...
2022-12-01T10:43:19.3365924Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3366115Z   test_new_subgroups_overlap_not_allowed (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2022-12-01T10:43:19.3366120Z 
2022-12-01T10:43:19.3366310Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3366384Z Ran 1 test in 0.002s
2022-12-01T10:43:19.3366389Z 
2022-12-01T10:43:19.3366458Z OK (skipped=1)
2022-12-01T10:43:19.3366463Z 
2022-12-01T10:43:19.3366547Z Generating XML reports...
2022-12-01T10:43:19.3366884Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104043.xml
2022-12-01T10:43:19.3367114Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.3367386Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3367537Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3367829Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3367962Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3367967Z 
2022-12-01T10:43:19.3368037Z Running tests...
2022-12-01T10:43:19.3368230Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3368451Z   test_new_subgroups_world_size_not_divisible_by_group_size (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2022-12-01T10:43:19.3368456Z 
2022-12-01T10:43:19.3368646Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3368719Z Ran 1 test in 0.002s
2022-12-01T10:43:19.3368724Z 
2022-12-01T10:43:19.3368793Z OK (skipped=1)
2022-12-01T10:43:19.3368798Z 
2022-12-01T10:43:19.3368881Z Generating XML reports...
2022-12-01T10:43:19.3369206Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104044.xml
2022-12-01T10:43:19.3369439Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.3369710Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3369832Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3370112Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3370245Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3370250Z 
2022-12-01T10:43:19.3370320Z Running tests...
2022-12-01T10:43:19.3370515Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3371231Z   test_output_unused_in_loss_dict_module (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78112 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.239s)
2022-12-01T10:43:19.3371239Z 
2022-12-01T10:43:19.3371430Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3371492Z Ran 1 test in 0.239s
2022-12-01T10:43:19.3371497Z 
2022-12-01T10:43:19.3371566Z OK (skipped=1)
2022-12-01T10:43:19.3371570Z 
2022-12-01T10:43:19.3371651Z Generating XML reports...
2022-12-01T10:43:19.3371988Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104046.xml
2022-12-01T10:43:19.3372218Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.3372497Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3372623Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3372902Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3373022Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3373041Z 
2022-12-01T10:43:19.3373100Z Running tests...
2022-12-01T10:43:19.3373377Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3373628Z   test_output_unused_in_loss_tuple_module (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4547
2022-12-01T10:43:19.3373785Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4548
2022-12-01T10:43:19.3373937Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4549
2022-12-01T10:43:19.3374249Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3374435Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3374713Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3374834Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3375013Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:43:19.3375285Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3375407Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3375685Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3375823Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3376000Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:43:19.3376269Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3376378Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3376654Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3376788Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3376964Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:43:19.3377260Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.3377556Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.3377849Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.3378015Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:43:19.3378177Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:43:19.3378326Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:43:19.3378429Z skip: Need at least 2 CUDA devices (1.855s)
2022-12-01T10:43:19.3378435Z 
2022-12-01T10:43:19.3378631Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3378704Z Ran 1 test in 1.855s
2022-12-01T10:43:19.3378710Z 
2022-12-01T10:43:19.3378780Z OK (skipped=1)
2022-12-01T10:43:19.3378785Z 
2022-12-01T10:43:19.3378867Z Generating XML reports...
2022-12-01T10:43:19.3379208Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104047.xml
2022-12-01T10:43:19.3379441Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.3379715Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3379824Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3380141Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3380274Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3380278Z 
2022-12-01T10:43:19.3380458Z Running tests...
2022-12-01T10:43:19.3380682Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3380979Z   test_periodic_model_averager (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4600
2022-12-01T10:43:19.3381181Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4601
2022-12-01T10:43:19.3381330Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4602
2022-12-01T10:43:19.3381605Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3381715Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3381996Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3382130Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3382308Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:43:19.3382581Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3382705Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3382988Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3383123Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3383287Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:43:19.3383558Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3383680Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3383960Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3384094Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3384271Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:43:19.3384569Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.3384866Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.3385158Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.3385309Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:43:19.3385467Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:43:19.3385627Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:43:19.3385733Z skip: Need at least 2 CUDA devices (1.856s)
2022-12-01T10:43:19.3385740Z 
2022-12-01T10:43:19.3385939Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3386015Z Ran 1 test in 1.856s
2022-12-01T10:43:19.3386020Z 
2022-12-01T10:43:19.3386091Z OK (skipped=1)
2022-12-01T10:43:19.3386095Z 
2022-12-01T10:43:19.3386179Z Generating XML reports...
2022-12-01T10:43:19.3386520Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104051.xml
2022-12-01T10:43:19.3386738Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.3387013Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3387139Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3387419Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3387581Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3387586Z 
2022-12-01T10:43:19.3387685Z Running tests...
2022-12-01T10:43:19.3387878Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3388127Z   test_periodic_model_averager_param_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4653
2022-12-01T10:43:19.3388269Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4654
2022-12-01T10:43:19.3388422Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4655
2022-12-01T10:43:19.3388695Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3388818Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3389098Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3389235Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3389413Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:43:19.3389683Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3389806Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3390073Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3390207Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3390384Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:43:19.3390653Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3390779Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3391056Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3391193Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3391369Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:43:19.3391668Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.3391953Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.3392250Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.3392414Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:43:19.3392579Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:43:19.3392742Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:43:19.3392845Z skip: Need at least 2 CUDA devices (1.857s)
2022-12-01T10:43:19.3392850Z 
2022-12-01T10:43:19.3393044Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3393117Z Ran 1 test in 1.857s
2022-12-01T10:43:19.3393122Z 
2022-12-01T10:43:19.3393179Z OK (skipped=1)
2022-12-01T10:43:19.3393198Z 
2022-12-01T10:43:19.3393299Z Generating XML reports...
2022-12-01T10:43:19.3393639Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104054.xml
2022-12-01T10:43:19.3393869Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.3394141Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3394298Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3394605Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3394738Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3394743Z 
2022-12-01T10:43:19.3394817Z Running tests...
2022-12-01T10:43:19.3394998Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3395703Z   test_post_localSGD_optimizer_parity (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77123 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.240s)
2022-12-01T10:43:19.3395722Z 
2022-12-01T10:43:19.3395905Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3395980Z Ran 1 test in 0.240s
2022-12-01T10:43:19.3395987Z 
2022-12-01T10:43:19.3396056Z OK (skipped=1)
2022-12-01T10:43:19.3396061Z 
2022-12-01T10:43:19.3396146Z Generating XML reports...
2022-12-01T10:43:19.3396484Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104057.xml
2022-12-01T10:43:19.3396716Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.3396986Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3397110Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3397378Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3397514Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3397521Z 
2022-12-01T10:43:19.3397592Z Running tests...
2022-12-01T10:43:19.3397787Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3398503Z   test_post_localSGD_optimizer_parity_grad_is_view (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77292 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.235s)
2022-12-01T10:43:19.3398509Z 
2022-12-01T10:43:19.3398701Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3398775Z Ran 1 test in 0.235s
2022-12-01T10:43:19.3398779Z 
2022-12-01T10:43:19.3398851Z OK (skipped=1)
2022-12-01T10:43:19.3398855Z 
2022-12-01T10:43:19.3398943Z Generating XML reports...
2022-12-01T10:43:19.3399282Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104059.xml
2022-12-01T10:43:19.3399502Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.3399773Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3399896Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3400173Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3400306Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3400311Z 
2022-12-01T10:43:19.3400382Z Running tests...
2022-12-01T10:43:19.3400574Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3400881Z   test_post_localSGD_optimizer_parity_with_hierarchical_sgd (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4726
2022-12-01T10:43:19.3401037Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4727
2022-12-01T10:43:19.3401212Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4728
2022-12-01T10:43:19.3401480Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3401602Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3401882Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3402018Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3402195Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:43:19.3402490Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.3402759Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3402872Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3403151Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3403284Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3403462Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:43:19.3403733Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3403856Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3404139Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3404275Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3404455Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:43:19.3404741Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.3405041Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.3405207Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:43:19.3405375Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:43:19.3405538Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:43:19.3405640Z skip: Need at least 4 CUDA devices (1.755s)
2022-12-01T10:43:19.3405646Z 
2022-12-01T10:43:19.3405843Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3405919Z Ran 1 test in 1.755s
2022-12-01T10:43:19.3405924Z 
2022-12-01T10:43:19.3405994Z OK (skipped=1)
2022-12-01T10:43:19.3405999Z 
2022-12-01T10:43:19.3406069Z Generating XML reports...
2022-12-01T10:43:19.3406406Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104101.xml
2022-12-01T10:43:19.3406641Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.3406912Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3407038Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3407318Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3407452Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3407488Z 
2022-12-01T10:43:19.3407562Z Running tests...
2022-12-01T10:43:19.3407769Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3408048Z   test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4779
2022-12-01T10:43:19.3408203Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4780
2022-12-01T10:43:19.3408357Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4781
2022-12-01T10:43:19.3408627Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3408750Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3409032Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3409167Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3409345Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:43:19.3409602Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3409725Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3410005Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3410141Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3410315Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:43:19.3410584Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3410710Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3410985Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3411121Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3411284Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:43:19.3411579Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.3411877Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.3412169Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.3412331Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:43:19.3412498Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:43:19.3412661Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:43:19.3412767Z skip: Need at least 4 CUDA devices (1.754s)
2022-12-01T10:43:19.3412772Z 
2022-12-01T10:43:19.3412963Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3413025Z Ran 1 test in 1.754s
2022-12-01T10:43:19.3413030Z 
2022-12-01T10:43:19.3413100Z OK (skipped=1)
2022-12-01T10:43:19.3413105Z 
2022-12-01T10:43:19.3413188Z Generating XML reports...
2022-12-01T10:43:19.3413617Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104104.xml
2022-12-01T10:43:19.3413850Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.3414160Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3414287Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3414597Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3414719Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3414736Z 
2022-12-01T10:43:19.3414796Z Running tests...
2022-12-01T10:43:19.3414992Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3415702Z   test_post_localSGD_optimizer_step_reload (__main__.TestDistBackendWithSpawn) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/84886 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.233s)
2022-12-01T10:43:19.3415708Z 
2022-12-01T10:43:19.3415905Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3415983Z Ran 1 test in 0.233s
2022-12-01T10:43:19.3415987Z 
2022-12-01T10:43:19.3416057Z OK (skipped=1)
2022-12-01T10:43:19.3416062Z 
2022-12-01T10:43:19.3416144Z Generating XML reports...
2022-12-01T10:43:19.3416483Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104107.xml
2022-12-01T10:43:19.3416714Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.3416975Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3417099Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3417378Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3417512Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3417519Z 
2022-12-01T10:43:19.3417592Z Running tests...
2022-12-01T10:43:19.3417789Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3418023Z   test_reduce_full_group_max (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4842
2022-12-01T10:43:19.3418179Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4843
2022-12-01T10:43:19.3418331Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4844
2022-12-01T10:43:19.3418587Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3418710Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3418990Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3419126Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3419303Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:43:19.3419574Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3419696Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3419972Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3420107Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3420270Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:43:19.3420709Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3420836Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3421176Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3421346Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3421521Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:43:19.3421823Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.3422121Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.3422398Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.3422564Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:43:19.3422731Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:43:19.3422896Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:43:19.3423071Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T10:43:19.3423244Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T10:43:19.3423415Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-12-01T10:43:19.3423705Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:43:19.3423998Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:43:19.3424284Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:43:19.3424520Z STAGE:2022-12-01 10:41:10 4842:4842 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.3424764Z STAGE:2022-12-01 10:41:10 4844:4844 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.3425005Z STAGE:2022-12-01 10:41:10 4843:4843 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.3425253Z STAGE:2022-12-01 10:41:10 4844:4844 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.3425492Z STAGE:2022-12-01 10:41:10 4843:4843 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.3425735Z STAGE:2022-12-01 10:41:10 4842:4842 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.3426129Z STAGE:2022-12-01 10:41:10 4843:4843 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-12-01 10:41:10 4844:4844 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.3426136Z 
2022-12-01T10:43:19.3426373Z STAGE:2022-12-01 10:41:10 4842:4842 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.3426619Z STAGE:2022-12-01 10:41:10 4844:4844 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.3426846Z STAGE:2022-12-01 10:41:10 4842:4842 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.3427082Z STAGE:2022-12-01 10:41:10 4843:4843 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.3427473Z STAGE:2022-12-01 10:41:10 4844:4844 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-12-01 10:41:10 4842:4842 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.3427478Z 
2022-12-01T10:43:19.3427721Z STAGE:2022-12-01 10:41:10 4843:4843 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.3427958Z STAGE:2022-12-01 10:41:10 4843:4843 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.3428225Z STAGE:2022-12-01 10:41:10 4842:4842 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.3428504Z STAGE:2022-12-01 10:41:10 4844:4844 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.3428571Z ok (1.954s)
2022-12-01T10:43:19.3428577Z 
2022-12-01T10:43:19.3428771Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3428834Z Ran 1 test in 1.954s
2022-12-01T10:43:19.3428850Z 
2022-12-01T10:43:19.3428898Z OK
2022-12-01T10:43:19.3428902Z 
2022-12-01T10:43:19.3428988Z Generating XML reports...
2022-12-01T10:43:19.3429327Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104109.xml
2022-12-01T10:43:19.3429559Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.3429835Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3429960Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3430240Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3430375Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3430380Z 
2022-12-01T10:43:19.3430440Z Running tests...
2022-12-01T10:43:19.3430634Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3430869Z   test_reduce_full_group_min (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4907
2022-12-01T10:43:19.3431024Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4908
2022-12-01T10:43:19.3431177Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4909
2022-12-01T10:43:19.3431450Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3431572Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3431853Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3431989Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3432156Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:43:19.3432425Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3432549Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3432830Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3432964Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3433141Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:43:19.3433411Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3433532Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3433796Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3433929Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3434102Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:43:19.3434400Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.3434698Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.3435017Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.3435213Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:43:19.3435374Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:43:19.3435536Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:43:19.3435697Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T10:43:19.3435869Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-12-01T10:43:19.3436039Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T10:43:19.3436331Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:43:19.3436626Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:43:19.3436915Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:43:19.3437162Z STAGE:2022-12-01 10:41:14 4907:4907 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.3437562Z STAGE:2022-12-01 10:41:14 4909:4909 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-12-01 10:41:14 4908:4908 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.3437567Z 
2022-12-01T10:43:19.3437812Z STAGE:2022-12-01 10:41:14 4909:4909 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.3438052Z STAGE:2022-12-01 10:41:14 4908:4908 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.3438279Z STAGE:2022-12-01 10:41:14 4907:4907 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.3438519Z STAGE:2022-12-01 10:41:14 4909:4909 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.3438759Z STAGE:2022-12-01 10:41:14 4908:4908 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.3438999Z STAGE:2022-12-01 10:41:14 4907:4907 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.3439238Z STAGE:2022-12-01 10:41:14 4909:4909 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.3439471Z STAGE:2022-12-01 10:41:14 4907:4907 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.3439714Z STAGE:2022-12-01 10:41:14 4908:4908 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.3439947Z STAGE:2022-12-01 10:41:14 4909:4909 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.3440183Z STAGE:2022-12-01 10:41:14 4907:4907 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.3440409Z STAGE:2022-12-01 10:41:14 4908:4908 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.3440644Z STAGE:2022-12-01 10:41:14 4908:4908 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.3440885Z STAGE:2022-12-01 10:41:14 4907:4907 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.3441121Z STAGE:2022-12-01 10:41:14 4909:4909 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.3441187Z ok (2.055s)
2022-12-01T10:43:19.3441192Z 
2022-12-01T10:43:19.3441388Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3441463Z Ran 1 test in 2.055s
2022-12-01T10:43:19.3441468Z 
2022-12-01T10:43:19.3441526Z OK
2022-12-01T10:43:19.3441531Z 
2022-12-01T10:43:19.3441602Z Generating XML reports...
2022-12-01T10:43:19.3441975Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104112.xml
2022-12-01T10:43:19.3442233Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.3442511Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3442637Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3442916Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3443052Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3443057Z 
2022-12-01T10:43:19.3443128Z Running tests...
2022-12-01T10:43:19.3443323Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3443552Z   test_reduce_full_group_product (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4972
2022-12-01T10:43:19.3443707Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4973
2022-12-01T10:43:19.3443861Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4974
2022-12-01T10:43:19.3444132Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3444256Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3444535Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3444671Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3444846Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:43:19.3445116Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3445229Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3445512Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3445647Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3445823Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:43:19.3446093Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3446216Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3446493Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3446627Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3446791Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:43:19.3447089Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.3447391Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.3447684Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.3447849Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:43:19.3448015Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:43:19.3448178Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:43:19.3448352Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T10:43:19.3448558Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T10:43:19.3448745Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-12-01T10:43:19.3449044Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:43:19.3449342Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:43:19.3449632Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:43:19.3449881Z STAGE:2022-12-01 10:41:17 4972:4972 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.3450281Z STAGE:2022-12-01 10:41:17 4974:4974 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-12-01 10:41:17 4973:4973 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.3450288Z 
2022-12-01T10:43:19.3450696Z STAGE:2022-12-01 10:41:17 4972:4972 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 10:41:17 4974:4974 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.3450945Z STAGE:2022-12-01 10:41:17 4973:4973 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.3450951Z 
2022-12-01T10:43:19.3451343Z STAGE:2022-12-01 10:41:17 4973:4973 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-12-01 10:41:17 4974:4974 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.3451349Z 
2022-12-01T10:43:19.3451587Z STAGE:2022-12-01 10:41:17 4972:4972 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.3451833Z STAGE:2022-12-01 10:41:17 4974:4974 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.3452076Z STAGE:2022-12-01 10:41:17 4972:4972 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.3452301Z STAGE:2022-12-01 10:41:17 4973:4973 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.3452539Z STAGE:2022-12-01 10:41:17 4974:4974 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.3452777Z STAGE:2022-12-01 10:41:17 4972:4972 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.3453010Z STAGE:2022-12-01 10:41:17 4973:4973 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.3453322Z STAGE:2022-12-01 10:41:17 4972:4972 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.3453573Z STAGE:2022-12-01 10:41:17 4973:4973 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.3453810Z STAGE:2022-12-01 10:41:17 4974:4974 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.3453879Z ok (1.955s)
2022-12-01T10:43:19.3453884Z 
2022-12-01T10:43:19.3454068Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3454147Z Ran 1 test in 1.955s
2022-12-01T10:43:19.3454151Z 
2022-12-01T10:43:19.3454213Z OK
2022-12-01T10:43:19.3454218Z 
2022-12-01T10:43:19.3454303Z Generating XML reports...
2022-12-01T10:43:19.3454646Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104116.xml
2022-12-01T10:43:19.3454879Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.3455154Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3455277Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3455558Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3455682Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3455720Z 
2022-12-01T10:43:19.3455794Z Running tests...
2022-12-01T10:43:19.3456024Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3456260Z   test_reduce_full_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5037
2022-12-01T10:43:19.3456415Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5038
2022-12-01T10:43:19.3456570Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5039
2022-12-01T10:43:19.3456845Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3456969Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3457234Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3457371Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3457549Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:43:19.3457821Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3457943Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3458219Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3458354Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3458532Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:43:19.3458800Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3458909Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3459187Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3459324Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3459504Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:43:19.3459804Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.3460099Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.3460493Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.3460663Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:43:19.3460825Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:43:19.3460977Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:43:19.3461154Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T10:43:19.3461330Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-12-01T10:43:19.3461628Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:43:19.3461800Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T10:43:19.3462090Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:43:19.3462384Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:43:19.3462677Z STAGE:2022-12-01 10:41:21 5037:5037 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.3462956Z STAGE:2022-12-01 10:41:21 5039:5039 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.3463185Z STAGE:2022-12-01 10:41:21 5038:5038 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.3463434Z STAGE:2022-12-01 10:41:21 5039:5039 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.3463677Z STAGE:2022-12-01 10:41:21 5038:5038 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.3463915Z STAGE:2022-12-01 10:41:21 5037:5037 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.3464156Z STAGE:2022-12-01 10:41:21 5038:5038 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.3464393Z STAGE:2022-12-01 10:41:21 5039:5039 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.3464634Z STAGE:2022-12-01 10:41:21 5037:5037 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.3464877Z STAGE:2022-12-01 10:41:21 5039:5039 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.3465113Z STAGE:2022-12-01 10:41:21 5037:5037 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.3465344Z STAGE:2022-12-01 10:41:21 5038:5038 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.3465581Z STAGE:2022-12-01 10:41:21 5039:5039 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.3465818Z STAGE:2022-12-01 10:41:21 5037:5037 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.3466055Z STAGE:2022-12-01 10:41:21 5038:5038 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.3466295Z STAGE:2022-12-01 10:41:21 5038:5038 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.3466537Z STAGE:2022-12-01 10:41:21 5037:5037 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.3466775Z STAGE:2022-12-01 10:41:21 5039:5039 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.3466842Z ok (1.857s)
2022-12-01T10:43:19.3466848Z 
2022-12-01T10:43:19.3467034Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3467108Z Ran 1 test in 1.857s
2022-12-01T10:43:19.3467113Z 
2022-12-01T10:43:19.3467172Z OK
2022-12-01T10:43:19.3467177Z 
2022-12-01T10:43:19.3467259Z Generating XML reports...
2022-12-01T10:43:19.3467601Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104119.xml
2022-12-01T10:43:19.3467834Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.3468108Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3468237Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3468522Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3468644Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3468649Z 
2022-12-01T10:43:19.3468721Z Running tests...
2022-12-01T10:43:19.3468916Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3469149Z   test_reduce_group_max (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5102
2022-12-01T10:43:19.3469302Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5103
2022-12-01T10:43:19.3469453Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5104
2022-12-01T10:43:19.3469761Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3469885Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3470185Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3470318Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3470495Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:43:19.3470764Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3470886Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3471165Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3471300Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3471478Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:43:19.3471746Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3471857Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3472137Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3472269Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3472441Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:43:19.3472738Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.3473032Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.3473329Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.3473498Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:43:19.3473663Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:43:19.3473822Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-12-01T10:43:19.3473992Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T10:43:19.3474154Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:43:19.3474322Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T10:43:19.3474615Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:43:19.3474909Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:43:19.3475201Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:43:19.3475446Z STAGE:2022-12-01 10:41:24 5103:5103 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.3475688Z STAGE:2022-12-01 10:41:24 5104:5104 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.3475923Z STAGE:2022-12-01 10:41:24 5104:5104 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.3476164Z STAGE:2022-12-01 10:41:24 5103:5103 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.3476403Z STAGE:2022-12-01 10:41:24 5104:5104 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.3476673Z STAGE:2022-12-01 10:41:24 5103:5103 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.3476958Z STAGE:2022-12-01 10:41:24 5104:5104 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.3477198Z STAGE:2022-12-01 10:41:24 5103:5103 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.3477267Z ok (1.956s)
2022-12-01T10:43:19.3477272Z 
2022-12-01T10:43:19.3477465Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3477539Z Ran 1 test in 1.956s
2022-12-01T10:43:19.3477544Z 
2022-12-01T10:43:19.3477592Z OK
2022-12-01T10:43:19.3477597Z 
2022-12-01T10:43:19.3477679Z Generating XML reports...
2022-12-01T10:43:19.3478020Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104122.xml
2022-12-01T10:43:19.3478250Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.3478525Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3478653Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3478936Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3479074Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3479079Z 
2022-12-01T10:43:19.3479138Z Running tests...
2022-12-01T10:43:19.3479334Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3479564Z   test_reduce_group_min (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5163
2022-12-01T10:43:19.3479718Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5164
2022-12-01T10:43:19.3479868Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5165
2022-12-01T10:43:19.3480147Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3480273Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3480553Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3480687Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3480851Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:43:19.3481122Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3481242Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3481518Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3481654Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3481831Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:43:19.3482100Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3482222Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3482486Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3482619Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3482798Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:43:19.3483094Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.3483474Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.3483795Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.3483957Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:43:19.3484123Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:43:19.3484284Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:43:19.3484446Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T10:43:19.3484620Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-12-01T10:43:19.3484790Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T10:43:19.3485085Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:43:19.3485380Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:43:19.3485629Z STAGE:2022-12-01 10:41:28 5164:5164 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.3485919Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:43:19.3486162Z STAGE:2022-12-01 10:41:28 5165:5165 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.3486409Z STAGE:2022-12-01 10:41:28 5165:5165 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.3486640Z STAGE:2022-12-01 10:41:28 5164:5164 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.3486881Z STAGE:2022-12-01 10:41:28 5165:5165 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.3487118Z STAGE:2022-12-01 10:41:28 5164:5164 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.3487363Z STAGE:2022-12-01 10:41:28 5164:5164 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.3487602Z STAGE:2022-12-01 10:41:28 5165:5165 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.3487670Z ok (2.057s)
2022-12-01T10:43:19.3487676Z 
2022-12-01T10:43:19.3487869Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3487944Z Ran 1 test in 2.057s
2022-12-01T10:43:19.3487949Z 
2022-12-01T10:43:19.3488013Z OK
2022-12-01T10:43:19.3488017Z 
2022-12-01T10:43:19.3488089Z Generating XML reports...
2022-12-01T10:43:19.3488428Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104126.xml
2022-12-01T10:43:19.3488664Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.3488938Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3489065Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3489348Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3489483Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3489487Z 
2022-12-01T10:43:19.3489557Z Running tests...
2022-12-01T10:43:19.3489742Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3489976Z   test_reduce_group_product (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5224
2022-12-01T10:43:19.3490130Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5225
2022-12-01T10:43:19.3490317Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5226
2022-12-01T10:43:19.3490614Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3490736Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3491013Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3491151Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3491327Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:43:19.3491582Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3491702Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3491979Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3492111Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3492286Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:43:19.3492554Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3492675Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3492948Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3493085Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3493317Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:43:19.3493633Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.3493930Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.3494222Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.3494387Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:43:19.3494547Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:43:19.3494711Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:43:19.3494883Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T10:43:19.3495057Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-12-01T10:43:19.3495219Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T10:43:19.3495512Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:43:19.3495807Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:43:19.3496050Z STAGE:2022-12-01 10:41:31 5225:5225 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.3496344Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:43:19.3496581Z STAGE:2022-12-01 10:41:31 5226:5226 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.3496829Z STAGE:2022-12-01 10:41:31 5226:5226 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.3497073Z STAGE:2022-12-01 10:41:31 5225:5225 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.3497353Z STAGE:2022-12-01 10:41:31 5226:5226 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.3497608Z STAGE:2022-12-01 10:41:31 5225:5225 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.3497851Z STAGE:2022-12-01 10:41:31 5225:5225 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.3498089Z STAGE:2022-12-01 10:41:31 5226:5226 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.3498158Z ok (1.959s)
2022-12-01T10:43:19.3498165Z 
2022-12-01T10:43:19.3498362Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3498438Z Ran 1 test in 1.959s
2022-12-01T10:43:19.3498443Z 
2022-12-01T10:43:19.3498502Z OK
2022-12-01T10:43:19.3498507Z 
2022-12-01T10:43:19.3498591Z Generating XML reports...
2022-12-01T10:43:19.3498917Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104129.xml
2022-12-01T10:43:19.3499159Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.3499434Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3499558Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3499839Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3499976Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3499981Z 
2022-12-01T10:43:19.3500052Z Running tests...
2022-12-01T10:43:19.3500248Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3500607Z   test_reduce_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5285
2022-12-01T10:43:19.3500752Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5286
2022-12-01T10:43:19.3500907Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5287
2022-12-01T10:43:19.3501182Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3501306Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3501588Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3501725Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3501902Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:43:19.3502202Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.3502473Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3502582Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3502861Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3502996Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3503171Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:43:19.3503440Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3503565Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3503843Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3503976Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3504189Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:43:19.3504526Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.3504817Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.3504986Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:43:19.3505149Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:43:19.3505310Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:43:19.3505483Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T10:43:19.3505658Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T10:43:19.3505832Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-12-01T10:43:19.3506115Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:43:19.3506409Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:43:19.3506654Z STAGE:2022-12-01 10:41:34 5286:5286 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.3506897Z STAGE:2022-12-01 10:41:34 5287:5287 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.3507156Z STAGE:2022-12-01 10:41:34 5287:5287 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.3507394Z STAGE:2022-12-01 10:41:34 5286:5286 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.3507636Z STAGE:2022-12-01 10:41:34 5287:5287 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.3507870Z STAGE:2022-12-01 10:41:34 5286:5286 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.3508117Z STAGE:2022-12-01 10:41:34 5286:5286 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.3508341Z STAGE:2022-12-01 10:41:34 5287:5287 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.3508637Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:43:19.3508705Z ok (1.954s)
2022-12-01T10:43:19.3508710Z 
2022-12-01T10:43:19.3508905Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3508982Z Ran 1 test in 1.954s
2022-12-01T10:43:19.3508987Z 
2022-12-01T10:43:19.3509049Z OK
2022-12-01T10:43:19.3509053Z 
2022-12-01T10:43:19.3509139Z Generating XML reports...
2022-12-01T10:43:19.3509480Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104133.xml
2022-12-01T10:43:19.3509714Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.3509974Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3510096Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3510375Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3510514Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3510519Z 
2022-12-01T10:43:19.3510591Z Running tests...
2022-12-01T10:43:19.3510786Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3511009Z   test_reduce_max (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5346
2022-12-01T10:43:19.3511196Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5347
2022-12-01T10:43:19.3511363Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5348
2022-12-01T10:43:19.3511635Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3511758Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3512035Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3512170Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3512345Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:43:19.3512616Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3512742Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3513022Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3513145Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3513322Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:43:19.3513590Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3513714Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3513992Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3514126Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3514305Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:43:19.3514604Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.3514905Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.3515185Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.3515351Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:43:19.3515597Z STAGE:2022-12-01 10:41:38 5347:5347 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.3515758Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:43:19.3515923Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:43:19.3516173Z STAGE:2022-12-01 10:41:38 5346:5346 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.3516418Z STAGE:2022-12-01 10:41:38 5348:5348 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.3516662Z STAGE:2022-12-01 10:41:38 5348:5348 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.3516891Z STAGE:2022-12-01 10:41:38 5346:5346 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.3517126Z STAGE:2022-12-01 10:41:38 5347:5347 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.3517364Z STAGE:2022-12-01 10:41:38 5348:5348 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.3517600Z STAGE:2022-12-01 10:41:38 5347:5347 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.3517833Z STAGE:2022-12-01 10:41:38 5346:5346 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.3518101Z STAGE:2022-12-01 10:41:38 5348:5348 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.3518365Z STAGE:2022-12-01 10:41:38 5346:5346 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.3518598Z STAGE:2022-12-01 10:41:38 5347:5347 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.3518838Z STAGE:2022-12-01 10:41:38 5348:5348 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.3519063Z STAGE:2022-12-01 10:41:38 5346:5346 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.3519296Z STAGE:2022-12-01 10:41:38 5347:5347 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.3519532Z STAGE:2022-12-01 10:41:38 5346:5346 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.3519767Z STAGE:2022-12-01 10:41:38 5347:5347 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.3520010Z STAGE:2022-12-01 10:41:38 5348:5348 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.3520081Z ok (1.958s)
2022-12-01T10:43:19.3520086Z 
2022-12-01T10:43:19.3520280Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3520355Z Ran 1 test in 1.958s
2022-12-01T10:43:19.3520360Z 
2022-12-01T10:43:19.3520420Z OK
2022-12-01T10:43:19.3520425Z 
2022-12-01T10:43:19.3520495Z Generating XML reports...
2022-12-01T10:43:19.3520840Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104136.xml
2022-12-01T10:43:19.3521072Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.3521343Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3521466Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3521750Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3521889Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3521893Z 
2022-12-01T10:43:19.3521964Z Running tests...
2022-12-01T10:43:19.3522144Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3522367Z   test_reduce_min (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5402
2022-12-01T10:43:19.3522520Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5403
2022-12-01T10:43:19.3522672Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5404
2022-12-01T10:43:19.3522945Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3523068Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3523351Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3523489Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3523666Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:43:19.3523923Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3524047Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3524325Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3524457Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3524635Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:43:19.3524933Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3525103Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3525381Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3525502Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3525674Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:43:19.3525973Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.3526270Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.3526562Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.3526729Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:43:19.3526975Z STAGE:2022-12-01 10:41:41 5403:5403 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.3527138Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:43:19.3527304Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:43:19.3527532Z STAGE:2022-12-01 10:41:41 5402:5402 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.3527775Z STAGE:2022-12-01 10:41:41 5404:5404 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.3528022Z STAGE:2022-12-01 10:41:41 5404:5404 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.3528431Z STAGE:2022-12-01 10:41:41 5402:5402 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 10:41:41 5403:5403 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.3528437Z 
2022-12-01T10:43:19.3528677Z STAGE:2022-12-01 10:41:41 5404:5404 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.3528916Z STAGE:2022-12-01 10:41:41 5403:5403 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.3529151Z STAGE:2022-12-01 10:41:41 5402:5402 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.3529396Z STAGE:2022-12-01 10:41:41 5404:5404 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.3529635Z STAGE:2022-12-01 10:41:41 5402:5402 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.3529877Z STAGE:2022-12-01 10:41:41 5403:5403 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.3530101Z STAGE:2022-12-01 10:41:41 5402:5402 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.3530340Z STAGE:2022-12-01 10:41:41 5404:5404 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.3530577Z STAGE:2022-12-01 10:41:41 5403:5403 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.3530814Z STAGE:2022-12-01 10:41:41 5402:5402 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.3531054Z STAGE:2022-12-01 10:41:41 5404:5404 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.3531291Z STAGE:2022-12-01 10:41:41 5403:5403 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.3531360Z ok (1.954s)
2022-12-01T10:43:19.3531365Z 
2022-12-01T10:43:19.3531558Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3531620Z Ran 1 test in 1.954s
2022-12-01T10:43:19.3531637Z 
2022-12-01T10:43:19.3531684Z OK
2022-12-01T10:43:19.3531689Z 
2022-12-01T10:43:19.3531773Z Generating XML reports...
2022-12-01T10:43:19.3532146Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104140.xml
2022-12-01T10:43:19.3532408Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.3532679Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3532805Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3533085Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3533218Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3533223Z 
2022-12-01T10:43:19.3533359Z Running tests...
2022-12-01T10:43:19.3533559Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3533761Z   test_reduce_multigpu (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl backend supports reduce multigpu (0.002s)
2022-12-01T10:43:19.3533766Z 
2022-12-01T10:43:19.3533960Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3534037Z Ran 1 test in 0.002s
2022-12-01T10:43:19.3534042Z 
2022-12-01T10:43:19.3534113Z OK (skipped=1)
2022-12-01T10:43:19.3534118Z 
2022-12-01T10:43:19.3534203Z Generating XML reports...
2022-12-01T10:43:19.3534541Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104143.xml
2022-12-01T10:43:19.3534775Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.3535035Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3535159Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3535438Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3535573Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3535581Z 
2022-12-01T10:43:19.3535654Z Running tests...
2022-12-01T10:43:19.3535847Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3536076Z   test_reduce_product (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5468
2022-12-01T10:43:19.3536234Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5469
2022-12-01T10:43:19.3536373Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5470
2022-12-01T10:43:19.3536646Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3536768Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3537046Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3537181Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3537359Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:43:19.3537628Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3537749Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3538028Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3538148Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3538328Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:43:19.3538597Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3538759Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3539037Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3539200Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3539375Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:43:19.3539677Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.3539972Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.3540252Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.3540529Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:43:19.3540777Z STAGE:2022-12-01 10:41:46 5470:5470 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.3540938Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:43:19.3541093Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:43:19.3541334Z STAGE:2022-12-01 10:41:46 5468:5468 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.3541576Z STAGE:2022-12-01 10:41:46 5469:5469 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.3541823Z STAGE:2022-12-01 10:41:46 5469:5469 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.3542065Z STAGE:2022-12-01 10:41:46 5470:5470 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.3542295Z STAGE:2022-12-01 10:41:46 5468:5468 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.3542536Z STAGE:2022-12-01 10:41:46 5469:5469 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.3542774Z STAGE:2022-12-01 10:41:46 5470:5470 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.3543011Z STAGE:2022-12-01 10:41:46 5468:5468 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.3543251Z STAGE:2022-12-01 10:41:46 5470:5470 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.3543660Z STAGE:2022-12-01 10:41:46 5469:5469 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 10:41:46 5468:5468 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.3543666Z 
2022-12-01T10:43:19.3543904Z STAGE:2022-12-01 10:41:46 5470:5470 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.3544145Z STAGE:2022-12-01 10:41:46 5468:5468 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.3544381Z STAGE:2022-12-01 10:41:46 5469:5469 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.3544610Z STAGE:2022-12-01 10:41:46 5469:5469 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.3545014Z STAGE:2022-12-01 10:41:46 5470:5470 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 10:41:46 5468:5468 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.3545019Z 
2022-12-01T10:43:19.3545086Z ok (1.954s)
2022-12-01T10:43:19.3545091Z 
2022-12-01T10:43:19.3545287Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3545362Z Ran 1 test in 1.955s
2022-12-01T10:43:19.3545367Z 
2022-12-01T10:43:19.3545427Z OK
2022-12-01T10:43:19.3545431Z 
2022-12-01T10:43:19.3545517Z Generating XML reports...
2022-12-01T10:43:19.3545857Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104144.xml
2022-12-01T10:43:19.3546151Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.3546449Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3546574Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3546857Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3546992Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3546997Z 
2022-12-01T10:43:19.3547070Z Running tests...
2022-12-01T10:43:19.3547263Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3547477Z   test_reduce_scatter_tensor_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA reduce_scatter_tensor (0.002s)
2022-12-01T10:43:19.3547482Z 
2022-12-01T10:43:19.3547677Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3547750Z Ran 1 test in 0.002s
2022-12-01T10:43:19.3547757Z 
2022-12-01T10:43:19.3547815Z OK (skipped=1)
2022-12-01T10:43:19.3547832Z 
2022-12-01T10:43:19.3547903Z Generating XML reports...
2022-12-01T10:43:19.3548244Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104148.xml
2022-12-01T10:43:19.3548476Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.3548753Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3548877Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3549158Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3549291Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3549296Z 
2022-12-01T10:43:19.3549371Z Running tests...
2022-12-01T10:43:19.3549552Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3549754Z   test_reduce_scatter_v_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports reduce_scatter_v (0.002s)
2022-12-01T10:43:19.3549760Z 
2022-12-01T10:43:19.3549950Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3550023Z Ran 1 test in 0.003s
2022-12-01T10:43:19.3550029Z 
2022-12-01T10:43:19.3550098Z OK (skipped=1)
2022-12-01T10:43:19.3550102Z 
2022-12-01T10:43:19.3550183Z Generating XML reports...
2022-12-01T10:43:19.3550522Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104149.xml
2022-12-01T10:43:19.3550747Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.3551021Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3551130Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3551410Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3551545Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3551550Z 
2022-12-01T10:43:19.3551620Z Running tests...
2022-12-01T10:43:19.3551811Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3552034Z   test_reduce_sum (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5544
2022-12-01T10:43:19.3552191Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5545
2022-12-01T10:43:19.3552344Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5546
2022-12-01T10:43:19.3552639Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3552789Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3553070Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3553203Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3553380Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:43:19.3553649Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3553771Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3554048Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3554180Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3554347Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:43:19.3554647Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.3554913Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3555035Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3555311Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3555442Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3555617Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:43:19.3555912Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.3556211Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.3556367Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:43:19.3556529Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:43:19.3556685Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:43:19.3556938Z STAGE:2022-12-01 10:41:52 5544:5544 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.3557180Z STAGE:2022-12-01 10:41:52 5545:5545 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.3557418Z STAGE:2022-12-01 10:41:52 5546:5546 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.3557664Z STAGE:2022-12-01 10:41:52 5546:5546 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.3557909Z STAGE:2022-12-01 10:41:52 5545:5545 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.3558136Z STAGE:2022-12-01 10:41:52 5544:5544 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.3558528Z STAGE:2022-12-01 10:41:52 5546:5546 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-12-01 10:41:52 5545:5545 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.3558533Z 
2022-12-01T10:43:19.3558769Z STAGE:2022-12-01 10:41:52 5544:5544 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.3559009Z STAGE:2022-12-01 10:41:52 5546:5546 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.3559246Z STAGE:2022-12-01 10:41:52 5545:5545 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.3559482Z STAGE:2022-12-01 10:41:52 5544:5544 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.3559748Z STAGE:2022-12-01 10:41:52 5546:5546 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.3560011Z STAGE:2022-12-01 10:41:52 5544:5544 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.3560247Z STAGE:2022-12-01 10:41:52 5545:5545 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.3560485Z STAGE:2022-12-01 10:41:52 5544:5544 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.3560707Z STAGE:2022-12-01 10:41:52 5545:5545 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.3560940Z STAGE:2022-12-01 10:41:52 5546:5546 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.3561007Z ok (1.856s)
2022-12-01T10:43:19.3561012Z 
2022-12-01T10:43:19.3561206Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3561281Z Ran 1 test in 1.856s
2022-12-01T10:43:19.3561286Z 
2022-12-01T10:43:19.3561347Z OK
2022-12-01T10:43:19.3561351Z 
2022-12-01T10:43:19.3561436Z Generating XML reports...
2022-12-01T10:43:19.3561775Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104151.xml
2022-12-01T10:43:19.3561995Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.3562271Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3562394Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3562674Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3562807Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3562812Z 
2022-12-01T10:43:19.3562883Z Running tests...
2022-12-01T10:43:19.3563079Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3563265Z   test_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA reduce (0.002s)
2022-12-01T10:43:19.3563273Z 
2022-12-01T10:43:19.3563464Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3563525Z Ran 1 test in 0.002s
2022-12-01T10:43:19.3563530Z 
2022-12-01T10:43:19.3563603Z OK (skipped=1)
2022-12-01T10:43:19.3563607Z 
2022-12-01T10:43:19.3563688Z Generating XML reports...
2022-12-01T10:43:19.3564022Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104154.xml
2022-12-01T10:43:19.3564254Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.3564531Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3564655Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3564936Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3565074Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3565079Z 
2022-12-01T10:43:19.3565139Z Running tests...
2022-12-01T10:43:19.3565335Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3565526Z   test_reduce_sum_cuda_twice (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA reduce (0.002s)
2022-12-01T10:43:19.3565531Z 
2022-12-01T10:43:19.3565721Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3565793Z Ran 1 test in 0.002s
2022-12-01T10:43:19.3565799Z 
2022-12-01T10:43:19.3565870Z OK (skipped=1)
2022-12-01T10:43:19.3565874Z 
2022-12-01T10:43:19.3565955Z Generating XML reports...
2022-12-01T10:43:19.3566326Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104155.xml
2022-12-01T10:43:19.3566556Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.3566840Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3566965Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3567243Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3567379Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3567383Z 
2022-12-01T10:43:19.3567454Z Running tests...
2022-12-01T10:43:19.3567649Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3567878Z   test_reduce_sum_twice (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5620
2022-12-01T10:43:19.3568037Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5621
2022-12-01T10:43:19.3568179Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5622
2022-12-01T10:43:19.3568451Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3568573Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3568851Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3568987Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3569164Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:43:19.3569433Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3569555Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3569832Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3569957Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3570135Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:43:19.3570406Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3570529Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3570808Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3570942Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3571117Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:43:19.3571415Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.3571715Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.3571993Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.3572155Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:43:19.3572401Z STAGE:2022-12-01 10:41:58 5621:5621 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.3572562Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:43:19.3572724Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:43:19.3572999Z STAGE:2022-12-01 10:41:58 5620:5620 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.3573241Z STAGE:2022-12-01 10:41:58 5622:5622 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.3573626Z STAGE:2022-12-01 10:41:58 5622:5622 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.3574034Z STAGE:2022-12-01 10:41:58 5620:5620 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 10:41:58 5621:5621 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.3574041Z 
2022-12-01T10:43:19.3574265Z STAGE:2022-12-01 10:41:58 5622:5622 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.3574504Z STAGE:2022-12-01 10:41:58 5621:5621 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.3574742Z STAGE:2022-12-01 10:41:58 5620:5620 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.3574988Z STAGE:2022-12-01 10:41:58 5620:5620 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.3575229Z STAGE:2022-12-01 10:41:58 5622:5622 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.3575473Z STAGE:2022-12-01 10:41:58 5621:5621 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.3575713Z STAGE:2022-12-01 10:41:58 5620:5620 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.3575947Z STAGE:2022-12-01 10:41:58 5622:5622 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.3576184Z STAGE:2022-12-01 10:41:58 5621:5621 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.3576576Z STAGE:2022-12-01 10:41:58 5621:5621 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 10:41:58 5620:5620 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.3576596Z 
2022-12-01T10:43:19.3576824Z STAGE:2022-12-01 10:41:58 5622:5622 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.3576894Z ok (1.954s)
2022-12-01T10:43:19.3576900Z 
2022-12-01T10:43:19.3577096Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3577172Z Ran 1 test in 1.954s
2022-12-01T10:43:19.3577176Z 
2022-12-01T10:43:19.3577236Z OK
2022-12-01T10:43:19.3577241Z 
2022-12-01T10:43:19.3577323Z Generating XML reports...
2022-12-01T10:43:19.3577666Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104157.xml
2022-12-01T10:43:19.3577897Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.3578158Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3578282Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3578564Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3578704Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3578711Z 
2022-12-01T10:43:19.3578784Z Running tests...
2022-12-01T10:43:19.3578983Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3579205Z   test_scatter (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5676
2022-12-01T10:43:19.3579361Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5677
2022-12-01T10:43:19.3579512Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5678
2022-12-01T10:43:19.3579771Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3579895Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3580211Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3580480Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3580660Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:43:19.3580936Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3581062Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3581347Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3581471Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3581650Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:43:19.3581920Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3582048Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3582333Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3582469Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3582644Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:43:19.3582942Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.3583237Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.3583519Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.3583686Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:43:19.3583849Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:43:19.3584096Z STAGE:2022-12-01 10:42:02 5677:5677 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.3584339Z STAGE:2022-12-01 10:42:02 5678:5678 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.3584500Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:43:19.3584737Z STAGE:2022-12-01 10:42:02 5676:5676 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.3584984Z STAGE:2022-12-01 10:42:02 5676:5676 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.3585223Z STAGE:2022-12-01 10:42:02 5677:5677 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.3585450Z STAGE:2022-12-01 10:42:02 5678:5678 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.3585686Z STAGE:2022-12-01 10:42:02 5676:5676 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.3585924Z STAGE:2022-12-01 10:42:02 5678:5678 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.3586162Z STAGE:2022-12-01 10:42:02 5677:5677 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.3586401Z STAGE:2022-12-01 10:42:02 5677:5677 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.3586638Z STAGE:2022-12-01 10:42:02 5676:5676 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.3586880Z STAGE:2022-12-01 10:42:02 5678:5678 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.3587115Z STAGE:2022-12-01 10:42:02 5676:5676 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.3587351Z STAGE:2022-12-01 10:42:02 5677:5677 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.3587623Z STAGE:2022-12-01 10:42:02 5678:5678 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.3587929Z STAGE:2022-12-01 10:42:02 5676:5676 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.3588169Z STAGE:2022-12-01 10:42:02 5678:5678 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.3588406Z STAGE:2022-12-01 10:42:02 5677:5677 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.3588477Z ok (1.954s)
2022-12-01T10:43:19.3588483Z 
2022-12-01T10:43:19.3588681Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3588757Z Ran 1 test in 1.954s
2022-12-01T10:43:19.3588762Z 
2022-12-01T10:43:19.3588822Z OK
2022-12-01T10:43:19.3588827Z 
2022-12-01T10:43:19.3588898Z Generating XML reports...
2022-12-01T10:43:19.3589243Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104200.xml
2022-12-01T10:43:19.3589480Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.3589756Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3589880Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3590162Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3590297Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3590302Z 
2022-12-01T10:43:19.3590374Z Running tests...
2022-12-01T10:43:19.3590568Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3590784Z   test_scatter_checks (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5732
2022-12-01T10:43:19.3590943Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5733
2022-12-01T10:43:19.3591096Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5734
2022-12-01T10:43:19.3591369Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3591495Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3591776Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3591912Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3592088Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:43:19.3592348Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3592469Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3592748Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3592884Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3593059Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:43:19.3593331Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3593455Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3593732Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3593867Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3594031Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:43:19.3594361Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.3594684Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.3594978Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.3595145Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:43:19.3595309Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:43:19.3595470Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:43:19.3595539Z ok (1.856s)
2022-12-01T10:43:19.3595544Z 
2022-12-01T10:43:19.3595739Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3595801Z Ran 1 test in 1.856s
2022-12-01T10:43:19.3595808Z 
2022-12-01T10:43:19.3595868Z OK
2022-12-01T10:43:19.3595873Z 
2022-12-01T10:43:19.3595956Z Generating XML reports...
2022-12-01T10:43:19.3596296Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104203.xml
2022-12-01T10:43:19.3596529Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.3596801Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3596925Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3597202Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3597324Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3597341Z 
2022-12-01T10:43:19.3597400Z Running tests...
2022-12-01T10:43:19.3597594Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3597821Z   test_scatter_complex (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5785
2022-12-01T10:43:19.3597976Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5786
2022-12-01T10:43:19.3598127Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5787
2022-12-01T10:43:19.3598399Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3598521Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3598797Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3598921Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3599099Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:43:19.3599370Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3599494Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3599771Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3599904Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3600080Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:43:19.3600350Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3600460Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3600734Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3600901Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3601105Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:43:19.3601405Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.3601701Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.3601989Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.3602154Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:43:19.3602317Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:43:19.3602470Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:43:19.3602715Z STAGE:2022-12-01 10:42:08 5786:5786 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.3602960Z STAGE:2022-12-01 10:42:08 5785:5785 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.3603200Z STAGE:2022-12-01 10:42:08 5787:5787 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.3603443Z STAGE:2022-12-01 10:42:08 5786:5786 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.3603851Z STAGE:2022-12-01 10:42:08 5785:5785 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 10:42:08 5787:5787 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.3603857Z 
2022-12-01T10:43:19.3604094Z STAGE:2022-12-01 10:42:08 5786:5786 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.3604330Z STAGE:2022-12-01 10:42:08 5787:5787 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.3604570Z STAGE:2022-12-01 10:42:08 5785:5785 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.3604814Z STAGE:2022-12-01 10:42:08 5787:5787 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.3605039Z STAGE:2022-12-01 10:42:08 5785:5785 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.3605273Z STAGE:2022-12-01 10:42:08 5786:5786 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.3605509Z STAGE:2022-12-01 10:42:08 5787:5787 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.3605743Z STAGE:2022-12-01 10:42:08 5785:5785 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.3605981Z STAGE:2022-12-01 10:42:08 5786:5786 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.3606224Z STAGE:2022-12-01 10:42:08 5785:5785 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.3606628Z STAGE:2022-12-01 10:42:08 5787:5787 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 10:42:08 5786:5786 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.3606636Z 
2022-12-01T10:43:19.3606702Z ok (1.852s)
2022-12-01T10:43:19.3606707Z 
2022-12-01T10:43:19.3606904Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3606966Z Ran 1 test in 1.852s
2022-12-01T10:43:19.3606983Z 
2022-12-01T10:43:19.3607030Z OK
2022-12-01T10:43:19.3607035Z 
2022-12-01T10:43:19.3607117Z Generating XML reports...
2022-12-01T10:43:19.3607457Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104207.xml
2022-12-01T10:43:19.3607689Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.3607993Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3608121Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3608428Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3608563Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3608568Z 
2022-12-01T10:43:19.3608628Z Running tests...
2022-12-01T10:43:19.3608824Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3609006Z   test_scatter_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA gather (0.001s)
2022-12-01T10:43:19.3609011Z 
2022-12-01T10:43:19.3609203Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3609278Z Ran 1 test in 0.002s
2022-12-01T10:43:19.3609282Z 
2022-12-01T10:43:19.3609352Z OK (skipped=1)
2022-12-01T10:43:19.3609356Z 
2022-12-01T10:43:19.3609438Z Generating XML reports...
2022-12-01T10:43:19.3609778Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104210.xml
2022-12-01T10:43:19.3610014Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.3610276Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3610398Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3610674Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3610807Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3610812Z 
2022-12-01T10:43:19.3610885Z Running tests...
2022-12-01T10:43:19.3611078Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3611273Z   test_scatter_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA gather (0.002s)
2022-12-01T10:43:19.3611280Z 
2022-12-01T10:43:19.3611469Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3611531Z Ran 1 test in 0.002s
2022-12-01T10:43:19.3611547Z 
2022-12-01T10:43:19.3611605Z OK (skipped=1)
2022-12-01T10:43:19.3611610Z 
2022-12-01T10:43:19.3611691Z Generating XML reports...
2022-12-01T10:43:19.3612027Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104211.xml
2022-12-01T10:43:19.3612256Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.3612526Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3612648Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3612930Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3613065Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3613072Z 
2022-12-01T10:43:19.3613130Z Running tests...
2022-12-01T10:43:19.3613425Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3613662Z   test_scatter_full_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5861
2022-12-01T10:43:19.3613817Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5862
2022-12-01T10:43:19.3613970Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5863
2022-12-01T10:43:19.3614245Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3614367Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3614681Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3614860Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3615026Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:43:19.3615295Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3615417Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3615691Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3615824Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3616001Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:43:19.3616271Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3616391Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3616659Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3616794Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3616968Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:43:19.3617267Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.3617560Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.3617853Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.3618023Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:43:19.3618189Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:43:19.3618353Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:43:19.3618515Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T10:43:19.3618687Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T10:43:19.3618857Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-12-01T10:43:19.3619146Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:43:19.3619439Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:43:19.3619725Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:43:19.3620130Z STAGE:2022-12-01 10:42:14 5861:5861 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-12-01 10:42:14 5863:5863 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.3620135Z 
2022-12-01T10:43:19.3620476Z STAGE:2022-12-01 10:42:14 5863:5863 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.3620719Z STAGE:2022-12-01 10:42:14 5863:5863 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.3620959Z STAGE:2022-12-01 10:42:14 5862:5862 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.3621191Z STAGE:2022-12-01 10:42:14 5862:5862 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.3621432Z STAGE:2022-12-01 10:42:14 5861:5861 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.3621719Z STAGE:2022-12-01 10:42:14 5862:5862 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.3622005Z STAGE:2022-12-01 10:42:14 5861:5861 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.3622248Z STAGE:2022-12-01 10:42:14 5863:5863 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.3622653Z STAGE:2022-12-01 10:42:14 5861:5861 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 10:42:14 5862:5862 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.3622659Z 
2022-12-01T10:43:19.3622897Z STAGE:2022-12-01 10:42:14 5862:5862 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.3623136Z STAGE:2022-12-01 10:42:14 5861:5861 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.3623373Z STAGE:2022-12-01 10:42:14 5863:5863 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.3623782Z STAGE:2022-12-01 10:42:14 5861:5861 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 10:42:14 5863:5863 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.3623789Z 
2022-12-01T10:43:19.3624020Z STAGE:2022-12-01 10:42:14 5862:5862 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.3624089Z ok (2.054s)
2022-12-01T10:43:19.3624095Z 
2022-12-01T10:43:19.3624289Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3624362Z Ran 1 test in 2.054s
2022-12-01T10:43:19.3624367Z 
2022-12-01T10:43:19.3624428Z OK
2022-12-01T10:43:19.3624433Z 
2022-12-01T10:43:19.3624517Z Generating XML reports...
2022-12-01T10:43:19.3624859Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104213.xml
2022-12-01T10:43:19.3625090Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.3625366Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3625480Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3625760Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3625894Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3625899Z 
2022-12-01T10:43:19.3625970Z Running tests...
2022-12-01T10:43:19.3626166Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3626391Z   test_scatter_group (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5926
2022-12-01T10:43:19.3626546Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5927
2022-12-01T10:43:19.3626699Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5928
2022-12-01T10:43:19.3626958Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3627084Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3627365Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3627500Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3627677Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:43:19.3627949Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3628074Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3628350Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3628512Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3628713Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:43:19.3628985Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3629109Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3629385Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3629520Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3629694Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:43:19.3629990Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.3630289Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.3630617Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.3630771Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:43:19.3630929Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:43:19.3631093Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:43:19.3631269Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T10:43:19.3631442Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-12-01T10:43:19.3631611Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T10:43:19.3631906Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:43:19.3632194Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:43:19.3632441Z STAGE:2022-12-01 10:42:18 5927:5927 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.3632736Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 3 nodes.
2022-12-01T10:43:19.3632975Z STAGE:2022-12-01 10:42:18 5928:5928 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.3633222Z STAGE:2022-12-01 10:42:18 5928:5928 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.3633460Z STAGE:2022-12-01 10:42:18 5927:5927 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.3633702Z STAGE:2022-12-01 10:42:18 5927:5927 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.3633931Z STAGE:2022-12-01 10:42:18 5928:5928 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.3634175Z STAGE:2022-12-01 10:42:18 5927:5927 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.3634412Z STAGE:2022-12-01 10:42:18 5928:5928 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.3634481Z ok (1.955s)
2022-12-01T10:43:19.3634487Z 
2022-12-01T10:43:19.3634683Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3634758Z Ran 1 test in 1.955s
2022-12-01T10:43:19.3634763Z 
2022-12-01T10:43:19.3634821Z OK
2022-12-01T10:43:19.3634826Z 
2022-12-01T10:43:19.3634909Z Generating XML reports...
2022-12-01T10:43:19.3635235Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104216.xml
2022-12-01T10:43:19.3635495Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.3635797Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3635922Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3636201Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3636336Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3636341Z 
2022-12-01T10:43:19.3636413Z Running tests...
2022-12-01T10:43:19.3636608Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3636837Z   test_scatter_object_list (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5987
2022-12-01T10:43:19.3636980Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5988
2022-12-01T10:43:19.3637135Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5989
2022-12-01T10:43:19.3637408Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3637529Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3637808Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3637941Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3638119Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:43:19.3638387Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3638496Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3638777Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3638912Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3639090Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:43:19.3639388Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.3639658Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3639781Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3640060Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3640194Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3640358Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:43:19.3640656Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.3640949Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.3641112Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:43:19.3641275Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:43:19.3641434Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:43:19.3641501Z ok (1.756s)
2022-12-01T10:43:19.3641506Z 
2022-12-01T10:43:19.3641700Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3641774Z Ran 1 test in 1.756s
2022-12-01T10:43:19.3641779Z 
2022-12-01T10:43:19.3641826Z OK
2022-12-01T10:43:19.3641861Z 
2022-12-01T10:43:19.3641946Z Generating XML reports...
2022-12-01T10:43:19.3642313Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104220.xml
2022-12-01T10:43:19.3642595Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.3642867Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3642990Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3643271Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3643405Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3643410Z 
2022-12-01T10:43:19.3643481Z Running tests...
2022-12-01T10:43:19.3643661Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3643883Z   test_send_recv (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6040
2022-12-01T10:43:19.3644038Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6041
2022-12-01T10:43:19.3644188Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6042
2022-12-01T10:43:19.3644458Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3644581Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3644860Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3644995Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3645159Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:43:19.3645430Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3645554Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3645832Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3645965Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3646139Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:43:19.3646409Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3646534Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3646810Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3646932Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3647106Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:43:19.3647405Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.3647700Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.3647987Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.3648151Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:43:19.3648313Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:43:19.3648477Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:43:19.3648532Z ok (1.855s)
2022-12-01T10:43:19.3648581Z 
2022-12-01T10:43:19.3648764Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3648866Z Ran 1 test in 1.855s
2022-12-01T10:43:19.3648870Z 
2022-12-01T10:43:19.3648929Z OK
2022-12-01T10:43:19.3648934Z 
2022-12-01T10:43:19.3649018Z Generating XML reports...
2022-12-01T10:43:19.3649357Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104223.xml
2022-12-01T10:43:19.3649589Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.3649861Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3649983Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3650248Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3650384Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3650391Z 
2022-12-01T10:43:19.3650462Z Running tests...
2022-12-01T10:43:19.3650657Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3650889Z   test_send_recv_any_source (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6093
2022-12-01T10:43:19.3651047Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6094
2022-12-01T10:43:19.3651199Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6095
2022-12-01T10:43:19.3651468Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3651589Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3651855Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3651990Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3652168Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:43:19.3652436Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3652558Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3652835Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3652967Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3653141Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:43:19.3653493Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3653621Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3653904Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3654041Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3654218Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:43:19.3654518Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.3654813Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.3655105Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.3655274Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:43:19.3655461Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:43:19.3655653Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:43:19.3655720Z ok (1.958s)
2022-12-01T10:43:19.3655725Z 
2022-12-01T10:43:19.3655920Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3655994Z Ran 1 test in 1.958s
2022-12-01T10:43:19.3656000Z 
2022-12-01T10:43:19.3656060Z OK
2022-12-01T10:43:19.3656065Z 
2022-12-01T10:43:19.3656147Z Generating XML reports...
2022-12-01T10:43:19.3656485Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104226.xml
2022-12-01T10:43:19.3656716Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.3656975Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3657100Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3657383Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3657522Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3657527Z 
2022-12-01T10:43:19.3657600Z Running tests...
2022-12-01T10:43:19.3657794Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3658047Z   test_send_recv_any_source_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6146
2022-12-01T10:43:19.3658202Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6147
2022-12-01T10:43:19.3658339Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6148
2022-12-01T10:43:19.3658616Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3658742Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3659020Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3659152Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3659331Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:43:19.3659600Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3659722Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3660000Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3660121Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3660300Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:43:19.3660671Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3660794Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3661075Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3661210Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3661387Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:43:19.3661687Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.3661984Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.3662321Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.3662524Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:43:19.3662689Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:43:19.3662939Z STAGE:2022-12-01 10:42:31 6147:6147 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.3663185Z STAGE:2022-12-01 10:42:31 6148:6148 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.3663350Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:43:19.3663590Z STAGE:2022-12-01 10:42:31 6146:6146 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.3663837Z STAGE:2022-12-01 10:42:31 6146:6146 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.3664068Z STAGE:2022-12-01 10:42:31 6147:6147 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.3664313Z STAGE:2022-12-01 10:42:31 6148:6148 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.3664382Z ok (1.952s)
2022-12-01T10:43:19.3664387Z 
2022-12-01T10:43:19.3664586Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3664663Z Ran 1 test in 1.953s
2022-12-01T10:43:19.3664669Z 
2022-12-01T10:43:19.3664732Z OK
2022-12-01T10:43:19.3664736Z 
2022-12-01T10:43:19.3664821Z Generating XML reports...
2022-12-01T10:43:19.3665164Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104230.xml
2022-12-01T10:43:19.3665398Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.3665663Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3665787Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3666071Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3666205Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3666211Z 
2022-12-01T10:43:19.3666282Z Running tests...
2022-12-01T10:43:19.3666476Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3666725Z   test_send_recv_any_source_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6202
2022-12-01T10:43:19.3666881Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6203
2022-12-01T10:43:19.3667033Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6204
2022-12-01T10:43:19.3667293Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3667419Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3667698Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3667833Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3668009Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:43:19.3668279Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3668402Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3668679Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3668800Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3669015Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:43:19.3669327Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3669450Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3669727Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3669861Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3670036Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:43:19.3670336Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.3670632Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.3670913Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.3671081Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:43:19.3671328Z STAGE:2022-12-01 10:42:35 6203:6203 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.3671491Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:43:19.3671647Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:43:19.3671888Z STAGE:2022-12-01 10:42:35 6202:6202 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.3672132Z STAGE:2022-12-01 10:42:35 6204:6204 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.3672382Z STAGE:2022-12-01 10:42:35 6202:6202 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.3672623Z STAGE:2022-12-01 10:42:35 6203:6203 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.3672855Z STAGE:2022-12-01 10:42:35 6204:6204 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.3672924Z ok (1.953s)
2022-12-01T10:43:19.3672929Z 
2022-12-01T10:43:19.3673125Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3673203Z Ran 1 test in 1.953s
2022-12-01T10:43:19.3673208Z 
2022-12-01T10:43:19.3673270Z OK
2022-12-01T10:43:19.3673274Z 
2022-12-01T10:43:19.3673359Z Generating XML reports...
2022-12-01T10:43:19.3673700Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104233.xml
2022-12-01T10:43:19.3673931Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.3674194Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3674320Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3674600Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3674736Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3674740Z 
2022-12-01T10:43:19.3674812Z Running tests...
2022-12-01T10:43:19.3675005Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3675246Z   test_send_recv_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6258
2022-12-01T10:43:19.3675401Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6259
2022-12-01T10:43:19.3675554Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6260
2022-12-01T10:43:19.3675849Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3675997Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3676278Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3676409Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3676588Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:43:19.3676858Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3676980Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3677256Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3677379Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3677557Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:43:19.3677828Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3677951Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3678228Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3678363Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3678540Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:43:19.3678838Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.3679135Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.3679419Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.3679589Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:43:19.3679753Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:43:19.3679998Z STAGE:2022-12-01 10:42:38 6259:6259 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.3680243Z STAGE:2022-12-01 10:42:38 6260:6260 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.3680405Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:43:19.3680648Z STAGE:2022-12-01 10:42:38 6258:6258 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.3680895Z STAGE:2022-12-01 10:42:38 6260:6260 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.3681134Z STAGE:2022-12-01 10:42:38 6258:6258 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.3681357Z STAGE:2022-12-01 10:42:38 6259:6259 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.3681426Z ok (1.871s)
2022-12-01T10:43:19.3681431Z 
2022-12-01T10:43:19.3681629Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3681705Z Ran 1 test in 1.872s
2022-12-01T10:43:19.3681711Z 
2022-12-01T10:43:19.3681771Z OK
2022-12-01T10:43:19.3681776Z 
2022-12-01T10:43:19.3681859Z Generating XML reports...
2022-12-01T10:43:19.3682199Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104237.xml
2022-12-01T10:43:19.3682433Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.3682738Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3682876Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3683156Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3683289Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3683294Z 
2022-12-01T10:43:19.3683367Z Running tests...
2022-12-01T10:43:19.3683560Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3683733Z   test_send_recv_nccl (__main__.TestDistBackendWithSpawn) ... skip: NCCL Send Recv Only (0.001s)
2022-12-01T10:43:19.3683738Z 
2022-12-01T10:43:19.3683932Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3684007Z Ran 1 test in 0.002s
2022-12-01T10:43:19.3684012Z 
2022-12-01T10:43:19.3684070Z OK (skipped=1)
2022-12-01T10:43:19.3684088Z 
2022-12-01T10:43:19.3684161Z Generating XML reports...
2022-12-01T10:43:19.3684502Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104240.xml
2022-12-01T10:43:19.3684737Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.3685011Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3685135Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3685418Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3685554Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3685559Z 
2022-12-01T10:43:19.3685632Z Running tests...
2022-12-01T10:43:19.3685813Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3686010Z   test_send_recv_nccl_autograd_profiler (__main__.TestDistBackendWithSpawn) ... skip: NCCL Send Recv Only (0.002s)
2022-12-01T10:43:19.3686017Z 
2022-12-01T10:43:19.3686209Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3686285Z Ran 1 test in 0.002s
2022-12-01T10:43:19.3686290Z 
2022-12-01T10:43:19.3686361Z OK (skipped=1)
2022-12-01T10:43:19.3686366Z 
2022-12-01T10:43:19.3686449Z Generating XML reports...
2022-12-01T10:43:19.3686787Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104241.xml
2022-12-01T10:43:19.3687020Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.3687294Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3687406Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3687687Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3687824Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3687829Z 
2022-12-01T10:43:19.3687905Z Running tests...
2022-12-01T10:43:19.3688099Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3688288Z   test_send_recv_nccl_torch_profiler (__main__.TestDistBackendWithSpawn) ... skip: NCCL Send Recv Only (0.002s)
2022-12-01T10:43:19.3688293Z 
2022-12-01T10:43:19.3688484Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3688559Z Ran 1 test in 0.002s
2022-12-01T10:43:19.3688563Z 
2022-12-01T10:43:19.3688621Z OK (skipped=1)
2022-12-01T10:43:19.3688637Z 
2022-12-01T10:43:19.3688708Z Generating XML reports...
2022-12-01T10:43:19.3689047Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104243.xml
2022-12-01T10:43:19.3689310Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.3689614Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3689736Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3690017Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3690150Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3690154Z 
2022-12-01T10:43:19.3690225Z Running tests...
2022-12-01T10:43:19.3690408Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3690645Z   test_send_recv_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6344
2022-12-01T10:43:19.3690798Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6345
2022-12-01T10:43:19.3690953Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6346
2022-12-01T10:43:19.3691228Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3691351Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3691631Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3691768Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3691944Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:43:19.3692202Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3692323Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3692604Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3692741Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3692918Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:43:19.3693188Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3693447Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3693730Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3693851Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3694027Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:43:19.3694327Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.3694625Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.3694919Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.3695085Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:43:19.3695333Z STAGE:2022-12-01 10:42:45 6345:6345 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.3695495Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:43:19.3695658Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:43:19.3695888Z STAGE:2022-12-01 10:42:45 6344:6344 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.3696196Z STAGE:2022-12-01 10:42:45 6346:6346 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.3696479Z STAGE:2022-12-01 10:42:45 6346:6346 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.3696723Z STAGE:2022-12-01 10:42:45 6344:6344 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.3696960Z STAGE:2022-12-01 10:42:45 6345:6345 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.3697030Z ok (1.955s)
2022-12-01T10:43:19.3697035Z 
2022-12-01T10:43:19.3697229Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3697306Z Ran 1 test in 1.956s
2022-12-01T10:43:19.3697311Z 
2022-12-01T10:43:19.3697375Z OK
2022-12-01T10:43:19.3697381Z 
2022-12-01T10:43:19.3697452Z Generating XML reports...
2022-12-01T10:43:19.3697793Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104244.xml
2022-12-01T10:43:19.3698027Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.3698307Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3698433Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3698716Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3698852Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3698858Z 
2022-12-01T10:43:19.3698930Z Running tests...
2022-12-01T10:43:19.3699113Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3699345Z   test_send_recv_with_tag (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6400
2022-12-01T10:43:19.3699501Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6401
2022-12-01T10:43:19.3699658Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6402
2022-12-01T10:43:19.3699935Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3700059Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3700339Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3700576Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3700752Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:43:19.3701010Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3701134Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3701416Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3701554Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3701732Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:43:19.3702031Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.3702300Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3702424Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3702703Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3702824Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3703049Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:43:19.3703387Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.3703683Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.3728372Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:43:19.3728637Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:43:19.3728804Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:43:19.3728872Z ok (1.751s)
2022-12-01T10:43:19.3728878Z 
2022-12-01T10:43:19.3729126Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3729197Z Ran 1 test in 1.752s
2022-12-01T10:43:19.3729203Z 
2022-12-01T10:43:19.3729252Z OK
2022-12-01T10:43:19.3729263Z 
2022-12-01T10:43:19.3729345Z Generating XML reports...
2022-12-01T10:43:19.3729696Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104247.xml
2022-12-01T10:43:19.3729931Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.3730209Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3730334Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3730620Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3730756Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3730762Z 
2022-12-01T10:43:19.3730835Z Running tests...
2022-12-01T10:43:19.3731019Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3731272Z   test_send_recv_with_tag_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6453
2022-12-01T10:43:19.3731432Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6454
2022-12-01T10:43:19.3731585Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6455
2022-12-01T10:43:19.3731858Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3731983Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3732266Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3732403Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3732570Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:43:19.3732847Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3732971Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3733349Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3733488Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3733667Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:43:19.3733939Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3734063Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3734343Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3734609Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3734822Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:43:19.3735121Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.3735417Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.3735709Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.3735872Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:43:19.3736120Z STAGE:2022-12-01 10:42:52 6455:6455 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.3736275Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:43:19.3736431Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:43:19.3736823Z STAGE:2022-12-01 10:42:52 6454:6454 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-12-01 10:42:52 6453:6453 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.3736834Z 
2022-12-01T10:43:19.3737068Z STAGE:2022-12-01 10:42:52 6453:6453 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.3737307Z STAGE:2022-12-01 10:42:52 6455:6455 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.3737539Z STAGE:2022-12-01 10:42:52 6454:6454 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.3737602Z ok (2.053s)
2022-12-01T10:43:19.3737606Z 
2022-12-01T10:43:19.3737812Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3737890Z Ran 1 test in 2.053s
2022-12-01T10:43:19.3737895Z 
2022-12-01T10:43:19.3737954Z OK
2022-12-01T10:43:19.3737959Z 
2022-12-01T10:43:19.3738040Z Generating XML reports...
2022-12-01T10:43:19.3738369Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104250.xml
2022-12-01T10:43:19.3738595Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.3738867Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3738989Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3739266Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3739399Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3739404Z 
2022-12-01T10:43:19.3739475Z Running tests...
2022-12-01T10:43:19.3739671Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3739917Z   test_send_recv_with_tag_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6509
2022-12-01T10:43:19.3740062Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6510
2022-12-01T10:43:19.3740213Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6511
2022-12-01T10:43:19.3740773Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3740890Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3741165Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3741291Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3741530Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:43:19.3741799Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3741963Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3742235Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3742362Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3742537Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:43:19.3742801Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3742919Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3743192Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3743322Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3743493Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:43:19.3743779Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.3744070Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.3744355Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.3744514Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:43:19.3744671Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:43:19.3744910Z STAGE:2022-12-01 10:42:55 6510:6510 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.3745145Z STAGE:2022-12-01 10:42:55 6511:6511 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.3745304Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:43:19.3745559Z STAGE:2022-12-01 10:42:56 6509:6509 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:43:19.3745792Z STAGE:2022-12-01 10:42:56 6509:6509 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.3746025Z STAGE:2022-12-01 10:42:56 6510:6510 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.3746256Z STAGE:2022-12-01 10:42:56 6511:6511 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:43:19.3746316Z ok (1.856s)
2022-12-01T10:43:19.3746321Z 
2022-12-01T10:43:19.3746509Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3746577Z Ran 1 test in 1.857s
2022-12-01T10:43:19.3746583Z 
2022-12-01T10:43:19.3746637Z OK
2022-12-01T10:43:19.3746641Z 
2022-12-01T10:43:19.3746721Z Generating XML reports...
2022-12-01T10:43:19.3747050Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104254.xml
2022-12-01T10:43:19.3747277Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.3747545Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3747661Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3747934Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3748061Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3748066Z 
2022-12-01T10:43:19.3748132Z Running tests...
2022-12-01T10:43:19.3748350Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3748584Z   test_sparse_all_reduce_sum (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6565
2022-12-01T10:43:19.3748757Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6566
2022-12-01T10:43:19.3748904Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6567
2022-12-01T10:43:19.3749168Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3749284Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3749557Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3749685Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3749858Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:43:19.3750121Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3750238Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3750504Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3750631Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3750801Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:43:19.3751096Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.3751359Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3751475Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3751748Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3751880Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3752043Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:43:19.3752334Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.3752620Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.3752777Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:43:19.3752934Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:43:19.3753090Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:43:19.3753150Z ok (1.755s)
2022-12-01T10:43:19.3753157Z 
2022-12-01T10:43:19.3753345Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3753413Z Ran 1 test in 1.756s
2022-12-01T10:43:19.3753418Z 
2022-12-01T10:43:19.3753465Z OK
2022-12-01T10:43:19.3753470Z 
2022-12-01T10:43:19.3753547Z Generating XML reports...
2022-12-01T10:43:19.3753879Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104257.xml
2022-12-01T10:43:19.3754105Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.3754374Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3754491Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3754794Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3754925Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3754960Z 
2022-12-01T10:43:19.3755026Z Running tests...
2022-12-01T10:43:19.3755208Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3755442Z   test_sparse_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6645
2022-12-01T10:43:19.3755591Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6646
2022-12-01T10:43:19.3755737Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6647
2022-12-01T10:43:19.3756003Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3756118Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3756394Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3756524Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3756687Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:43:19.3756949Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3757064Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3757335Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3757461Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3757629Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:43:19.3757895Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3758010Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3758283Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3758404Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3758571Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:43:19.3758862Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.3759154Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.3759439Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.3759598Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:43:19.3759755Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:43:19.3759909Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:43:19.3759998Z skip: CUDA is not available. (1.857s)
2022-12-01T10:43:19.3760003Z 
2022-12-01T10:43:19.3760185Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3760252Z Ran 1 test in 1.858s
2022-12-01T10:43:19.3760257Z 
2022-12-01T10:43:19.3760321Z OK (skipped=1)
2022-12-01T10:43:19.3760326Z 
2022-12-01T10:43:19.3760401Z Generating XML reports...
2022-12-01T10:43:19.3760736Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104300.xml
2022-12-01T10:43:19.3760960Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.3761258Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3761402Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3761669Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3761797Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3761802Z 
2022-12-01T10:43:19.3761867Z Running tests...
2022-12-01T10:43:19.3762054Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3762282Z   test_stateless_api_with_ddp (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6698
2022-12-01T10:43:19.3762428Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6699
2022-12-01T10:43:19.3762572Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6700
2022-12-01T10:43:19.3762840Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3762957Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3763224Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3763353Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3763522Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:43:19.3763788Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3763904Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3764175Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3764302Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3764476Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:43:19.3764742Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3764851Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3765122Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3765249Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3765418Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:43:19.3765708Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.3765998Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.3766284Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.3766443Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:43:19.3766598Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:43:19.3766747Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:43:19.3766843Z skip: Need at least 2 CUDA devices (1.855s)
2022-12-01T10:43:19.3766848Z 
2022-12-01T10:43:19.3767034Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3767103Z Ran 1 test in 1.855s
2022-12-01T10:43:19.3767109Z 
2022-12-01T10:43:19.3767170Z OK (skipped=1)
2022-12-01T10:43:19.3767175Z 
2022-12-01T10:43:19.3767252Z Generating XML reports...
2022-12-01T10:43:19.3767616Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104304.xml
2022-12-01T10:43:19.3767871Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.3768132Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3768247Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3768520Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3768649Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3768654Z 
2022-12-01T10:43:19.3768722Z Running tests...
2022-12-01T10:43:19.3768911Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3769140Z   test_static_graph_api_cpu (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6751
2022-12-01T10:43:19.3769289Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6752
2022-12-01T10:43:19.3769433Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6753
2022-12-01T10:43:19.3769690Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3769805Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3770076Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3770203Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3770371Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:43:19.3770637Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3770755Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3771029Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3771151Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3771318Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:43:19.3771581Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3771694Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3771965Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3772092Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3772262Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:43:19.3772556Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.3772845Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.3773126Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.3773361Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:43:19.3773519Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:43:19.3773698Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4yssz6nh
2022-12-01T10:43:19.3773931Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp1padiv3
2022-12-01T10:43:19.3774122Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4yssz6nh/_remote_module_non_scriptable.py
2022-12-01T10:43:19.3774343Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp1padiv3/_remote_module_non_scriptable.py
2022-12-01T10:43:19.3774501Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:43:19.3774679Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1d_b_09g
2022-12-01T10:43:19.3774856Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1d_b_09g/_remote_module_non_scriptable.py
2022-12-01T10:43:19.3774917Z ok (1.855s)
2022-12-01T10:43:19.3774923Z 
2022-12-01T10:43:19.3775114Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3775182Z Ran 1 test in 1.856s
2022-12-01T10:43:19.3775188Z 
2022-12-01T10:43:19.3775242Z OK
2022-12-01T10:43:19.3775246Z 
2022-12-01T10:43:19.3775322Z Generating XML reports...
2022-12-01T10:43:19.3775660Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104307.xml
2022-12-01T10:43:19.3775887Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.3776146Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3776265Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3776536Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3776663Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3776668Z 
2022-12-01T10:43:19.3776734Z Running tests...
2022-12-01T10:43:19.3776921Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3777142Z   test_sync_bn_logged (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6804
2022-12-01T10:43:19.3777293Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6805
2022-12-01T10:43:19.3777439Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6806
2022-12-01T10:43:19.3777697Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3777813Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3778080Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3778206Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3778374Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:43:19.3778640Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3778758Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3779033Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3779159Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3779322Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:43:19.3779581Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3779696Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3779968Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3780094Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3780295Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:43:19.3780743Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.3781035Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.3781314Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.3781471Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:43:19.3781624Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:43:19.3781777Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:43:19.3781874Z skip: Need at least 2 CUDA devices (1.756s)
2022-12-01T10:43:19.3781880Z 
2022-12-01T10:43:19.3782068Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3782137Z Ran 1 test in 1.756s
2022-12-01T10:43:19.3782142Z 
2022-12-01T10:43:19.3782206Z OK (skipped=1)
2022-12-01T10:43:19.3782211Z 
2022-12-01T10:43:19.3782287Z Generating XML reports...
2022-12-01T10:43:19.3782616Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104310.xml
2022-12-01T10:43:19.3782841Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.3783107Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3783223Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3783497Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3783628Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3783635Z 
2022-12-01T10:43:19.3783701Z Running tests...
2022-12-01T10:43:19.3783889Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3784142Z   test_undefined_grad_parity_unused_parameters (__main__.TestDistBackendWithSpawn) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6857
2022-12-01T10:43:19.3784285Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6858
2022-12-01T10:43:19.3784432Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6859
2022-12-01T10:43:19.3784696Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3784812Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3785084Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3785219Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3785389Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:43:19.3785658Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3785768Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3786041Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3786169Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3786340Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:43:19.3786630Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.3786945Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3787108Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3787379Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3787504Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3787665Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:43:19.3787955Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.3788244Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:43:19.3788403Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:43:19.3788561Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:43:19.3788717Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:43:19.3788815Z skip: Need at least 2 CUDA devices (1.753s)
2022-12-01T10:43:19.3788820Z 
2022-12-01T10:43:19.3789008Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3789077Z Ran 1 test in 1.754s
2022-12-01T10:43:19.3789082Z 
2022-12-01T10:43:19.3789140Z OK (skipped=1)
2022-12-01T10:43:19.3789145Z 
2022-12-01T10:43:19.3789221Z Generating XML reports...
2022-12-01T10:43:19.3789552Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104314.xml
2022-12-01T10:43:19.3789778Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.3790051Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3790169Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3790444Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3790573Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3790578Z 
2022-12-01T10:43:19.3790643Z Running tests...
2022-12-01T10:43:19.3790824Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3791155Z   test_verify_model_across_rank_with_logger (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'gloo', 'nccl'} (0.001s)
2022-12-01T10:43:19.3791161Z 
2022-12-01T10:43:19.3791344Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3791412Z Ran 1 test in 0.002s
2022-12-01T10:43:19.3791417Z 
2022-12-01T10:43:19.3791483Z OK (skipped=1)
2022-12-01T10:43:19.3791487Z 
2022-12-01T10:43:19.3791566Z Generating XML reports...
2022-12-01T10:43:19.3791901Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104317.xml
2022-12-01T10:43:19.3792124Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:43:19.3792390Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:43:19.3792499Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:43:19.3792768Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:43:19.3792894Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:43:19.3792899Z 
2022-12-01T10:43:19.3792965Z Running tests...
2022-12-01T10:43:19.3793153Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3793526Z   test_verify_model_across_rank_without_logger (__main__.TestDistBackendWithSpawn) ... skip: Test requires backends to be available {'gloo', 'nccl'} (0.001s)
2022-12-01T10:43:19.3793558Z 
2022-12-01T10:43:19.3793750Z ----------------------------------------------------------------------
2022-12-01T10:43:19.3793825Z Ran 1 test in 0.002s
2022-12-01T10:43:19.3793830Z 
2022-12-01T10:43:19.3793887Z OK (skipped=1)
2022-12-01T10:43:19.3793904Z 
2022-12-01T10:43:19.3793974Z Generating XML reports...
2022-12-01T10:43:19.3794313Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104318.xml
2022-12-01T10:43:19.3794318Z 
2022-12-01T10:43:19.3794720Z ##[endgroup]
2022-12-01T10:43:19.3795077Z FINISHED PRINTING LOG FILE of distributed/test_distributed_spawn (/var/lib/jenkins/workspace/test/test-reports/distributed-test_distributed_spawn_z513gh_5)
2022-12-01T10:43:19.3795082Z 
2022-12-01T10:43:19.3795335Z Running distributed/rpc/cuda/test_tensorpipe_agent ... [2022-12-01 10:43:19.214573]
2022-12-01T10:43:19.3795744Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/rpc/cuda/test_tensorpipe_agent.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-12-01 10:43:19.214837]
2022-12-01T10:49:11.9218515Z 
2022-12-01T10:49:11.9218936Z Expand the folded group to see the log file of distributed/rpc/cuda/test_tensorpipe_agent
2022-12-01T10:49:11.9223759Z ##[group]PRINTING LOG FILE of distributed/rpc/cuda/test_tensorpipe_agent (/var/lib/jenkins/workspace/test/test-reports/distributed-rpc-cuda-test_tensorpipe_agent_jd46ql5s)
2022-12-01T10:49:11.9224548Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa5ylc4be
2022-12-01T10:49:11.9225174Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa5ylc4be/_remote_module_non_scriptable.py
2022-12-01T10:49:11.9225854Z <unittest.suite.TestSuite tests=[<__main__.TensorPipeCudaDdpComparisonTest testMethod=test_ddp_dist_autograd_local_vs_remote_gpu>]>
2022-12-01T10:49:11.9226392Z test_ddp_dist_autograd_local_vs_remote_gpu (__main__.TensorPipeCudaDdpComparisonTest)
2022-12-01T10:49:11.9227206Z <unittest.suite.TestSuite tests=[<__main__.TensorPipeCudaDistAutogradTest testMethod=test_gpu_simple>, <__main__.TensorPipeCudaDistAutogradTest testMethod=test_gpu_to_cpu_continuation>, <__main__.TensorPipeCudaDistAutogradTest testMethod=test_gpu_to_cpu_continuation_gpu_root>]>
2022-12-01T10:49:11.9228104Z test_gpu_simple (__main__.TensorPipeCudaDistAutogradTest)
2022-12-01T10:49:11.9228645Z test_gpu_to_cpu_continuation (__main__.TensorPipeCudaDistAutogradTest)
2022-12-01T10:49:11.9229153Z test_gpu_to_cpu_continuation_gpu_root (__main__.TensorPipeCudaDistAutogradTest)
2022-12-01T10:49:11.9230178Z <unittest.suite.TestSuite tests=[<__main__.TensorPipeCudaRemoteModuleTest testMethod=test_input_moved_to_cuda_device>, <__main__.TensorPipeCudaRemoteModuleTest testMethod=test_input_moved_to_cuda_device_script>, <__main__.TensorPipeCudaRemoteModuleTest testMethod=test_invalid_devices>, <__main__.TensorPipeCudaRemoteModuleTest testMethod=test_valid_device>]>
2022-12-01T10:49:11.9231153Z test_input_moved_to_cuda_device (__main__.TensorPipeCudaRemoteModuleTest)
2022-12-01T10:49:11.9231626Z test_input_moved_to_cuda_device_script (__main__.TensorPipeCudaRemoteModuleTest)
2022-12-01T10:49:11.9232038Z test_invalid_devices (__main__.TensorPipeCudaRemoteModuleTest)
2022-12-01T10:49:11.9232462Z test_valid_device (__main__.TensorPipeCudaRemoteModuleTest)
2022-12-01T10:49:11.9232981Z <unittest.suite.TestSuite tests=[<__main__.TensorPipeCudaRpcTest testMethod=test_profiler_remote_cuda>]>
2022-12-01T10:49:11.9233397Z test_profiler_remote_cuda (__main__.TensorPipeCudaRpcTest)
2022-12-01T10:49:11.9302932Z <unittest.suite.TestSuite tests=[<__main__.TensorPipePipeWithDDPTest testMethod=test_basic_gloo_ckpt_always>, <__main__.TensorPipePipeWithDDPTest testMethod=test_basic_gloo_ckpt_except_last>, <__main__.TensorPipePipeWithDDPTest testMethod=test_basic_gloo_ckpt_never>, <__main__.TensorPipePipeWithDDPTest testMethod=test_basic_gloo_ckpt_never_find_unused>, <__main__.TensorPipePipeWithDDPTest testMethod=test_basic_nccl_ckpt_always>, <__main__.TensorPipePipeWithDDPTest testMethod=test_basic_nccl_ckpt_except_last>, <__main__.TensorPipePipeWithDDPTest testMethod=test_basic_nccl_ckpt_never>, <__main__.TensorPipePipeWithDDPTest testMethod=test_basic_nccl_ckpt_never_find_unused>]>
2022-12-01T10:49:11.9304604Z test_basic_gloo_ckpt_always (__main__.TensorPipePipeWithDDPTest)
2022-12-01T10:49:11.9305162Z test_basic_gloo_ckpt_except_last (__main__.TensorPipePipeWithDDPTest)
2022-12-01T10:49:11.9305746Z test_basic_gloo_ckpt_never (__main__.TensorPipePipeWithDDPTest)
2022-12-01T10:49:11.9306315Z test_basic_gloo_ckpt_never_find_unused (__main__.TensorPipePipeWithDDPTest)
2022-12-01T10:49:11.9306897Z test_basic_nccl_ckpt_always (__main__.TensorPipePipeWithDDPTest)
2022-12-01T10:49:11.9307465Z test_basic_nccl_ckpt_except_last (__main__.TensorPipePipeWithDDPTest)
2022-12-01T10:49:11.9307776Z test_basic_nccl_ckpt_never (__main__.TensorPipePipeWithDDPTest)
2022-12-01T10:49:11.9308092Z test_basic_nccl_ckpt_never_find_unused (__main__.TensorPipePipeWithDDPTest)
2022-12-01T10:49:11.9319602Z <unittest.suite.TestSuite tests=[<__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_async_execution_nested_with_cuda_future>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_async_execution_with_cuda_future>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_cuda_future_callback_changes_devices>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_cuda_future_can_extract_cuda_sparse_tensor>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_cuda_future_can_extract_cuda_tensor>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_cuda_future_can_extract_custom_class_with_cuda_sparse_tensor>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_cuda_future_can_extract_custom_class_with_cuda_tensor>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_cuda_future_can_extract_list_with_cuda_sparse_tensor>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_cuda_future_can_extract_list_with_cuda_tensor>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_cuda_future_device_as_device>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_cuda_future_device_as_int>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_cuda_future_device_as_str>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_cuda_future_device_not_cuda>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_cuda_future_modify_tensor_inplace>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_cuda_future_replace_tensor>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_cuda_future_value_on_bad_device>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_custom_stream>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_custom_stream_multi>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_custom_stream_nested>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_custom_stream_nested_multi>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_cpu>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_cpu_to_gpu_default>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_cpu_to_gpu_non_default>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_default>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_default_to_non_default>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_1>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_2>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_3>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_4>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_5>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_6>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_7>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_8>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_self_1>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_self_2>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_self_3>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_self_4>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_self_5>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_self_6>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_self_7>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_self_8>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_non_default>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_non_default_to_default>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_to_cpu_default>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_to_cpu_non_default>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_gpu>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_in_options>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_invalid_max_local_device>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_invalid_max_remote_device>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_invalid_min_device>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_many_to_one>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_missing_config>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_missing_config_loop>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_missing_config_not_timeout>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_missing_config_remote>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_missing_config_remote_response>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_missing_config_response>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_missing_config_response_loop>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_multi_gpu>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_multi_gpu_self>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_one_to_many>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_remote>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_return_to_gpu>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_return_to_gpu_self>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_wrong_worker_name>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_mismatch>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_devices_option_mismatch>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_devices_option_mismatch_reverse>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_owner_rref_forward_synchronization1>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_owner_rref_forward_synchronization2>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_owner_rref_forward_synchronization3>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_owner_rref_forward_synchronization4>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_rref_as_arg_synchronization1>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_rref_as_arg_synchronization2>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_rref_as_arg_synchronization3>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_rref_as_arg_synchronization4>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_rref_as_arg_synchronization5>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_rref_forward_synchronization1>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_rref_forward_synchronization2>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_rref_forward_synchronization3>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_rref_forward_synchronization4>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_rref_to_here_synchronization1>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_rref_to_here_synchronization2>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_rref_to_here_synchronization3>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_rref_to_here_synchronization4>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_rref_with_unpickleable_attributes>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_tensor_view_as_return_value>]>
2022-12-01T10:49:11.9331885Z test_async_execution_nested_with_cuda_future (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-12-01T10:49:11.9332281Z test_async_execution_with_cuda_future (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-12-01T10:49:11.9332750Z test_cuda_future_callback_changes_devices (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-12-01T10:49:11.9333150Z test_cuda_future_can_extract_cuda_sparse_tensor (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-12-01T10:49:11.9333544Z test_cuda_future_can_extract_cuda_tensor (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-12-01T10:49:11.9333946Z test_cuda_future_can_extract_custom_class_with_cuda_sparse_tensor (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-12-01T10:49:11.9334379Z test_cuda_future_can_extract_custom_class_with_cuda_tensor (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-12-01T10:49:11.9334808Z test_cuda_future_can_extract_list_with_cuda_sparse_tensor (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-12-01T10:49:11.9335203Z test_cuda_future_can_extract_list_with_cuda_tensor (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-12-01T10:49:11.9335591Z test_cuda_future_device_as_device (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-12-01T10:49:11.9335960Z test_cuda_future_device_as_int (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-12-01T10:49:11.9336317Z test_cuda_future_device_as_str (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-12-01T10:49:11.9336668Z test_cuda_future_device_not_cuda (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-12-01T10:49:11.9337040Z test_cuda_future_modify_tensor_inplace (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-12-01T10:49:11.9337412Z test_cuda_future_replace_tensor (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-12-01T10:49:11.9337773Z test_cuda_future_value_on_bad_device (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-12-01T10:49:11.9338123Z test_custom_stream (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-12-01T10:49:11.9338466Z test_custom_stream_multi (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-12-01T10:49:11.9338814Z test_custom_stream_nested (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-12-01T10:49:11.9339158Z test_custom_stream_nested_multi (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-12-01T10:49:11.9339509Z test_device_map_cpu (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-12-01T10:49:11.9339869Z test_device_map_cpu_to_gpu_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-12-01T10:49:11.9340232Z test_device_map_cpu_to_gpu_non_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-12-01T10:49:11.9340879Z test_device_map_gpu_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-12-01T10:49:11.9341622Z test_device_map_gpu_default_to_non_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-12-01T10:49:11.9342258Z test_device_map_gpu_mixed_1 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-12-01T10:49:11.9342850Z test_device_map_gpu_mixed_2 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-12-01T10:49:11.9343434Z test_device_map_gpu_mixed_3 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-12-01T10:49:11.9343985Z test_device_map_gpu_mixed_4 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-12-01T10:49:11.9344610Z test_device_map_gpu_mixed_5 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-12-01T10:49:11.9345126Z test_device_map_gpu_mixed_6 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-12-01T10:49:11.9345660Z test_device_map_gpu_mixed_7 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-12-01T10:49:11.9346291Z test_device_map_gpu_mixed_8 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-12-01T10:49:11.9346852Z test_device_map_gpu_mixed_self_1 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-12-01T10:49:11.9347520Z test_device_map_gpu_mixed_self_2 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-12-01T10:49:11.9348179Z test_device_map_gpu_mixed_self_3 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-12-01T10:49:11.9348783Z test_device_map_gpu_mixed_self_4 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-12-01T10:49:11.9349345Z test_device_map_gpu_mixed_self_5 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-12-01T10:49:11.9349907Z test_device_map_gpu_mixed_self_6 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-12-01T10:49:11.9350505Z test_device_map_gpu_mixed_self_7 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-12-01T10:49:11.9351078Z test_device_map_gpu_mixed_self_8 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-12-01T10:49:11.9351676Z test_device_map_gpu_non_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-12-01T10:49:11.9352301Z test_device_map_gpu_non_default_to_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-12-01T10:49:11.9352960Z test_device_map_gpu_to_cpu_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-12-01T10:49:11.9353620Z test_device_map_gpu_to_cpu_non_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-12-01T10:49:11.9354079Z test_device_maps_gpu (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-12-01T10:49:11.9354670Z test_device_maps_in_options (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-12-01T10:49:11.9355443Z test_device_maps_invalid_max_local_device (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-12-01T10:49:11.9356144Z test_device_maps_invalid_max_remote_device (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-12-01T10:49:11.9356781Z test_device_maps_invalid_min_device (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-12-01T10:49:11.9357148Z test_device_maps_many_to_one (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-12-01T10:49:11.9357511Z test_device_maps_missing_config (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-12-01T10:49:11.9357889Z test_device_maps_missing_config_loop (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-12-01T10:49:11.9358262Z test_device_maps_missing_config_not_timeout (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-12-01T10:49:11.9358644Z test_device_maps_missing_config_remote (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-12-01T10:49:11.9359034Z test_device_maps_missing_config_remote_response (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-12-01T10:49:11.9359413Z test_device_maps_missing_config_response (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-12-01T10:49:11.9359801Z test_device_maps_missing_config_response_loop (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-12-01T10:49:11.9360178Z test_device_maps_multi_gpu (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-12-01T10:49:11.9360535Z test_device_maps_multi_gpu_self (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-12-01T10:49:11.9360880Z test_device_maps_one_to_many (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-12-01T10:49:11.9361293Z test_device_maps_remote (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-12-01T10:49:11.9361685Z test_device_maps_return_to_gpu (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-12-01T10:49:11.9362038Z test_device_maps_return_to_gpu_self (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-12-01T10:49:11.9362412Z test_device_maps_wrong_worker_name (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-12-01T10:49:11.9362769Z test_device_mismatch (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-12-01T10:49:11.9363122Z test_devices_option_mismatch (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-12-01T10:49:11.9363475Z test_devices_option_mismatch_reverse (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-12-01T10:49:11.9363860Z test_owner_rref_forward_synchronization1 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-12-01T10:49:11.9364252Z test_owner_rref_forward_synchronization2 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-12-01T10:49:11.9364626Z test_owner_rref_forward_synchronization3 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-12-01T10:49:11.9365011Z test_owner_rref_forward_synchronization4 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-12-01T10:49:11.9365387Z test_rref_as_arg_synchronization1 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-12-01T10:49:11.9365755Z test_rref_as_arg_synchronization2 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-12-01T10:49:11.9366110Z test_rref_as_arg_synchronization3 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-12-01T10:49:11.9366472Z test_rref_as_arg_synchronization4 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-12-01T10:49:11.9366832Z test_rref_as_arg_synchronization5 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-12-01T10:49:11.9367192Z test_rref_forward_synchronization1 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-12-01T10:49:11.9367566Z test_rref_forward_synchronization2 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-12-01T10:49:11.9367940Z test_rref_forward_synchronization3 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-12-01T10:49:11.9368317Z test_rref_forward_synchronization4 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-12-01T10:49:11.9368673Z test_rref_to_here_synchronization1 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-12-01T10:49:11.9369043Z test_rref_to_here_synchronization2 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-12-01T10:49:11.9369411Z test_rref_to_here_synchronization3 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-12-01T10:49:11.9369763Z test_rref_to_here_synchronization4 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-12-01T10:49:11.9370134Z test_rref_with_unpickleable_attributes (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-12-01T10:49:11.9370506Z test_tensor_view_as_return_value (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-12-01T10:49:11.9371199Z <unittest.suite.TestSuite tests=[<__main__.TensorPipeTensorPipeCudaDistAutogradTest testMethod=test_device_maps_backward_pass>, <__main__.TensorPipeTensorPipeCudaDistAutogradTest testMethod=test_dist_autograd_sync_streams>, <__main__.TensorPipeTensorPipeCudaDistAutogradTest testMethod=test_gradients_synchronizations>]>
2022-12-01T10:49:11.9371881Z test_device_maps_backward_pass (__main__.TensorPipeTensorPipeCudaDistAutogradTest)
2022-12-01T10:49:11.9372255Z test_dist_autograd_sync_streams (__main__.TensorPipeTensorPipeCudaDistAutogradTest)
2022-12-01T10:49:11.9372722Z test_gradients_synchronizations (__main__.TensorPipeTensorPipeCudaDistAutogradTest)
2022-12-01T10:49:11.9373331Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:11.9373655Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:11.9374089Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:11.9374440Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:11.9374830Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn2xcq84a
2022-12-01T10:49:11.9375255Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn2xcq84a/_remote_module_non_scriptable.py
2022-12-01T10:49:11.9375740Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T10:49:11.9375961Z 
2022-12-01T10:49:11.9376036Z Running tests...
2022-12-01T10:49:11.9376339Z ----------------------------------------------------------------------
2022-12-01T10:49:11.9376734Z   test_ddp_dist_autograd_local_vs_remote_gpu (__main__.TensorPipeCudaDdpComparisonTest) ... skip: c10d was not compiled with the NCCL backend (0.002s)
2022-12-01T10:49:11.9376991Z 
2022-12-01T10:49:11.9377185Z ----------------------------------------------------------------------
2022-12-01T10:49:11.9377425Z Ran 1 test in 0.002s
2022-12-01T10:49:11.9377540Z 
2022-12-01T10:49:11.9377600Z OK (skipped=1)
2022-12-01T10:49:11.9377706Z 
2022-12-01T10:49:11.9377792Z Generating XML reports...
2022-12-01T10:49:11.9378306Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaDdpComparisonTest-20221201104321.xml
2022-12-01T10:49:11.9378891Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:11.9379209Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:11.9379635Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:11.9379981Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:11.9380322Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjn9vb_bw
2022-12-01T10:49:11.9381887Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjn9vb_bw/_remote_module_non_scriptable.py
2022-12-01T10:49:11.9382764Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T10:49:11.9383165Z 
2022-12-01T10:49:11.9383281Z Running tests...
2022-12-01T10:49:11.9383828Z ----------------------------------------------------------------------
2022-12-01T10:49:11.9384561Z   test_gpu_simple (__main__.TensorPipeCudaDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6950
2022-12-01T10:49:11.9385096Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6951
2022-12-01T10:49:11.9385628Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6952
2022-12-01T10:49:11.9386178Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 6953
2022-12-01T10:49:11.9386964Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:11.9387572Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:11.9388078Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:11.9388612Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:11.9389186Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_6n9mtg3
2022-12-01T10:49:11.9389854Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_6n9mtg3/_remote_module_non_scriptable.py
2022-12-01T10:49:11.9390697Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:11.9391292Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:11.9392086Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:11.9392461Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:11.9392893Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6t5s28h3
2022-12-01T10:49:11.9393300Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6t5s28h3/_remote_module_non_scriptable.py
2022-12-01T10:49:11.9393849Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:11.9394167Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:11.9394594Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:11.9394939Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:11.9395286Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcidpmtmi
2022-12-01T10:49:11.9395675Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcidpmtmi/_remote_module_non_scriptable.py
2022-12-01T10:49:11.9396166Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:11.9396496Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:11.9396910Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:11.9397254Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:11.9397597Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr956be70
2022-12-01T10:49:11.9397996Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr956be70/_remote_module_non_scriptable.py
2022-12-01T10:49:11.9398356Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:49:11.9398701Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:49:11.9399042Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:49:11.9399382Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:49:11.9399654Z skip: Need at least 1 CUDA device (1.854s)
2022-12-01T10:49:11.9399790Z 
2022-12-01T10:49:11.9399988Z ----------------------------------------------------------------------
2022-12-01T10:49:11.9400228Z Ran 1 test in 1.854s
2022-12-01T10:49:11.9400341Z 
2022-12-01T10:49:11.9400400Z OK (skipped=1)
2022-12-01T10:49:11.9400508Z 
2022-12-01T10:49:11.9400592Z Generating XML reports...
2022-12-01T10:49:11.9401107Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaDistAutogradTest-20221201104323.xml
2022-12-01T10:49:11.9401685Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:11.9401999Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:11.9402426Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:11.9402775Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:11.9403109Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc4q7cy_y
2022-12-01T10:49:11.9403511Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc4q7cy_y/_remote_module_non_scriptable.py
2022-12-01T10:49:11.9403996Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T10:49:11.9404212Z 
2022-12-01T10:49:11.9404285Z Running tests...
2022-12-01T10:49:11.9404576Z ----------------------------------------------------------------------
2022-12-01T10:49:11.9404990Z   test_gpu_to_cpu_continuation (__main__.TensorPipeCudaDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7005
2022-12-01T10:49:11.9405400Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7006
2022-12-01T10:49:11.9405766Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7007
2022-12-01T10:49:11.9406105Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 7008
2022-12-01T10:49:11.9406548Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:11.9406878Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:11.9407291Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:11.9407632Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:11.9407975Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr86c6q28
2022-12-01T10:49:11.9408378Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr86c6q28/_remote_module_non_scriptable.py
2022-12-01T10:49:11.9408846Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:11.9409176Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:11.9409606Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:11.9409936Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:11.9410278Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6ujazn73
2022-12-01T10:49:11.9410672Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6ujazn73/_remote_module_non_scriptable.py
2022-12-01T10:49:11.9411152Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:11.9411464Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:11.9411884Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:11.9412228Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:11.9412659Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1p87ojqb
2022-12-01T10:49:11.9413045Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1p87ojqb/_remote_module_non_scriptable.py
2022-12-01T10:49:11.9413529Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:11.9413858Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:11.9414270Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:11.9414613Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:11.9414959Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmf1jai43
2022-12-01T10:49:11.9415359Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmf1jai43/_remote_module_non_scriptable.py
2022-12-01T10:49:11.9415724Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:49:11.9416069Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:49:11.9416417Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:49:11.9416746Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:49:11.9417028Z skip: Need at least 1 CUDA device (1.758s)
2022-12-01T10:49:11.9417162Z 
2022-12-01T10:49:11.9417362Z ----------------------------------------------------------------------
2022-12-01T10:49:11.9417601Z Ran 1 test in 1.758s
2022-12-01T10:49:11.9417702Z 
2022-12-01T10:49:11.9417778Z OK (skipped=1)
2022-12-01T10:49:11.9417882Z 
2022-12-01T10:49:11.9417968Z Generating XML reports...
2022-12-01T10:49:11.9418520Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaDistAutogradTest-20221201104326.xml
2022-12-01T10:49:11.9419120Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:11.9419451Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:11.9419883Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:11.9420229Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:11.9420851Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpegep4tqf
2022-12-01T10:49:11.9421426Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpegep4tqf/_remote_module_non_scriptable.py
2022-12-01T10:49:11.9422235Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T10:49:11.9422574Z 
2022-12-01T10:49:11.9422667Z Running tests...
2022-12-01T10:49:11.9423147Z ----------------------------------------------------------------------
2022-12-01T10:49:11.9423869Z   test_gpu_to_cpu_continuation_gpu_root (__main__.TensorPipeCudaDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7060
2022-12-01T10:49:11.9424548Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7061
2022-12-01T10:49:11.9425073Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7062
2022-12-01T10:49:11.9425606Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 7063
2022-12-01T10:49:11.9426451Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:11.9426975Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:11.9427663Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:11.9428238Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:11.9428816Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyocir13a
2022-12-01T10:49:11.9429475Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyocir13a/_remote_module_non_scriptable.py
2022-12-01T10:49:11.9430349Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:11.9430745Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:11.9431169Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:11.9431502Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:11.9431846Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9fjb0j3s
2022-12-01T10:49:11.9432248Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9fjb0j3s/_remote_module_non_scriptable.py
2022-12-01T10:49:11.9432737Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:11.9433192Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:11.9433621Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:11.9433961Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:11.9434302Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7w9f6vpw
2022-12-01T10:49:11.9434684Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7w9f6vpw/_remote_module_non_scriptable.py
2022-12-01T10:49:11.9435173Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:11.9435614Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:11.9436034Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:11.9436428Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:11.9436768Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp11gx2j7d
2022-12-01T10:49:11.9437165Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp11gx2j7d/_remote_module_non_scriptable.py
2022-12-01T10:49:11.9437526Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:49:11.9437877Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:49:11.9438218Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:49:11.9438551Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:49:11.9438833Z skip: Need at least 1 CUDA device (1.953s)
2022-12-01T10:49:11.9438968Z 
2022-12-01T10:49:11.9439172Z ----------------------------------------------------------------------
2022-12-01T10:49:11.9439415Z Ran 1 test in 1.953s
2022-12-01T10:49:11.9439516Z 
2022-12-01T10:49:11.9439586Z OK (skipped=1)
2022-12-01T10:49:11.9439691Z 
2022-12-01T10:49:11.9439776Z Generating XML reports...
2022-12-01T10:49:11.9440292Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaDistAutogradTest-20221201104329.xml
2022-12-01T10:49:11.9440859Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:11.9441190Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:11.9441615Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:11.9441960Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:11.9442288Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5kwfl9lr
2022-12-01T10:49:11.9442692Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5kwfl9lr/_remote_module_non_scriptable.py
2022-12-01T10:49:11.9443172Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T10:49:11.9443389Z 
2022-12-01T10:49:11.9443461Z Running tests...
2022-12-01T10:49:11.9443746Z ----------------------------------------------------------------------
2022-12-01T10:49:11.9444171Z   test_input_moved_to_cuda_device (__main__.TensorPipeCudaRemoteModuleTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7115
2022-12-01T10:49:11.9444585Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7116
2022-12-01T10:49:11.9445019Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:11.9445346Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:11.9445773Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:11.9446121Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:11.9446456Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplulnh3r6
2022-12-01T10:49:11.9446912Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplulnh3r6/_remote_module_non_scriptable.py
2022-12-01T10:49:11.9447394Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:11.9447723Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:11.9448135Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:11.9448514Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:11.9448883Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9qrddh1n
2022-12-01T10:49:11.9449268Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9qrddh1n/_remote_module_non_scriptable.py
2022-12-01T10:49:11.9449653Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:49:11.9450000Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:49:11.9450282Z skip: Need at least 1 CUDA device (1.748s)
2022-12-01T10:49:11.9450403Z 
2022-12-01T10:49:11.9450603Z ----------------------------------------------------------------------
2022-12-01T10:49:11.9450840Z Ran 1 test in 1.748s
2022-12-01T10:49:11.9450954Z 
2022-12-01T10:49:11.9451030Z OK (skipped=1)
2022-12-01T10:49:11.9451137Z 
2022-12-01T10:49:11.9451207Z Generating XML reports...
2022-12-01T10:49:11.9451724Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaRemoteModuleTest-20221201104333.xml
2022-12-01T10:49:11.9452383Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:11.9452718Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:11.9453134Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:11.9453479Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:11.9453824Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvncf38x2
2022-12-01T10:49:11.9454215Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvncf38x2/_remote_module_non_scriptable.py
2022-12-01T10:49:11.9454698Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T10:49:11.9454918Z 
2022-12-01T10:49:11.9454992Z Running tests...
2022-12-01T10:49:11.9455293Z ----------------------------------------------------------------------
2022-12-01T10:49:11.9455708Z   test_input_moved_to_cuda_device_script (__main__.TensorPipeCudaRemoteModuleTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7148
2022-12-01T10:49:11.9456132Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7149
2022-12-01T10:49:11.9456584Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:11.9457034Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:11.9457646Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:11.9458306Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:11.9459068Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa5pdtsi1
2022-12-01T10:49:11.9459934Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa5pdtsi1/_remote_module_non_scriptable.py
2022-12-01T10:49:11.9461047Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:11.9461590Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:11.9462311Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:11.9462906Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:11.9463536Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2z0m8cyh
2022-12-01T10:49:11.9464281Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2z0m8cyh/_remote_module_non_scriptable.py
2022-12-01T10:49:11.9464983Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:49:11.9465675Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:49:11.9466156Z skip: Need at least 1 CUDA device (1.744s)
2022-12-01T10:49:11.9466367Z 
2022-12-01T10:49:11.9466709Z ----------------------------------------------------------------------
2022-12-01T10:49:11.9467086Z Ran 1 test in 1.744s
2022-12-01T10:49:11.9467274Z 
2022-12-01T10:49:11.9467391Z OK (skipped=1)
2022-12-01T10:49:11.9467571Z 
2022-12-01T10:49:11.9467713Z Generating XML reports...
2022-12-01T10:49:11.9468519Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaRemoteModuleTest-20221201104336.xml
2022-12-01T10:49:11.9469438Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:11.9470001Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:11.9470728Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:11.9471312Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:11.9471868Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdptsqoex
2022-12-01T10:49:11.9472547Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdptsqoex/_remote_module_non_scriptable.py
2022-12-01T10:49:11.9473367Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T10:49:11.9473724Z 
2022-12-01T10:49:11.9473827Z Running tests...
2022-12-01T10:49:11.9474333Z ----------------------------------------------------------------------
2022-12-01T10:49:11.9475055Z   test_invalid_devices (__main__.TensorPipeCudaRemoteModuleTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7181
2022-12-01T10:49:11.9475746Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7182
2022-12-01T10:49:11.9476478Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:11.9477083Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:11.9477799Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:11.9478380Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:11.9478943Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2cuzndk4
2022-12-01T10:49:11.9479642Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2cuzndk4/_remote_module_non_scriptable.py
2022-12-01T10:49:11.9480447Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:11.9480993Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:11.9481729Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:11.9482298Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:11.9482838Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo0g4d8ec
2022-12-01T10:49:11.9483475Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo0g4d8ec/_remote_module_non_scriptable.py
2022-12-01T10:49:11.9484154Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:49:11.9484740Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:49:11.9485214Z skip: Need at least 1 CUDA device (1.750s)
2022-12-01T10:49:11.9485431Z 
2022-12-01T10:49:11.9485754Z ----------------------------------------------------------------------
2022-12-01T10:49:11.9486160Z Ran 1 test in 1.751s
2022-12-01T10:49:11.9486347Z 
2022-12-01T10:49:11.9486470Z OK (skipped=1)
2022-12-01T10:49:11.9486651Z 
2022-12-01T10:49:11.9486875Z Generating XML reports...
2022-12-01T10:49:11.9487766Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaRemoteModuleTest-20221201104339.xml
2022-12-01T10:49:11.9488711Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:11.9489264Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:11.9489951Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:11.9490523Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:11.9491079Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptu57p9np
2022-12-01T10:49:11.9491722Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptu57p9np/_remote_module_non_scriptable.py
2022-12-01T10:49:11.9492586Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T10:49:11.9492946Z 
2022-12-01T10:49:11.9493066Z Running tests...
2022-12-01T10:49:11.9493594Z ----------------------------------------------------------------------
2022-12-01T10:49:11.9494266Z   test_valid_device (__main__.TensorPipeCudaRemoteModuleTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7214
2022-12-01T10:49:11.9494927Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7215
2022-12-01T10:49:11.9495691Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:11.9496238Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:11.9496939Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:11.9497498Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:11.9498072Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwq1e__th
2022-12-01T10:49:11.9498745Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwq1e__th/_remote_module_non_scriptable.py
2022-12-01T10:49:11.9499563Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:11.9500158Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:11.9501079Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:11.9501718Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:11.9502289Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7euchaty
2022-12-01T10:49:11.9502699Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7euchaty/_remote_module_non_scriptable.py
2022-12-01T10:49:11.9503079Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:49:11.9503433Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:49:11.9503707Z skip: Need at least 1 CUDA device (1.749s)
2022-12-01T10:49:11.9503842Z 
2022-12-01T10:49:11.9504043Z ----------------------------------------------------------------------
2022-12-01T10:49:11.9504302Z Ran 1 test in 1.749s
2022-12-01T10:49:11.9504413Z 
2022-12-01T10:49:11.9504473Z OK (skipped=1)
2022-12-01T10:49:11.9504582Z 
2022-12-01T10:49:11.9504666Z Generating XML reports...
2022-12-01T10:49:11.9505175Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaRemoteModuleTest-20221201104343.xml
2022-12-01T10:49:11.9505749Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:11.9506151Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:11.9506585Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:11.9506988Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:11.9507319Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9l3zfae9
2022-12-01T10:49:11.9507717Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9l3zfae9/_remote_module_non_scriptable.py
2022-12-01T10:49:11.9508198Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T10:49:11.9508416Z 
2022-12-01T10:49:11.9508488Z Running tests...
2022-12-01T10:49:11.9508780Z ----------------------------------------------------------------------
2022-12-01T10:49:11.9509178Z   test_profiler_remote_cuda (__main__.TensorPipeCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7247
2022-12-01T10:49:11.9509574Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7248
2022-12-01T10:49:11.9509893Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7249
2022-12-01T10:49:11.9510216Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 7250
2022-12-01T10:49:11.9510663Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:11.9510988Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:11.9511403Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:11.9511753Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:11.9512098Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpenuvet_d
2022-12-01T10:49:11.9512502Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpenuvet_d/_remote_module_non_scriptable.py
2022-12-01T10:49:11.9512977Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:11.9513310Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:11.9513738Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:11.9514067Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:11.9514409Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptprp2gr1
2022-12-01T10:49:11.9514811Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptprp2gr1/_remote_module_non_scriptable.py
2022-12-01T10:49:11.9515295Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:11.9515609Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:11.9516037Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:11.9516385Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:11.9516712Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplgfpapnk
2022-12-01T10:49:11.9517109Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplgfpapnk/_remote_module_non_scriptable.py
2022-12-01T10:49:11.9517592Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:11.9517918Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:11.9518328Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:11.9518665Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:11.9519040Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwpoi8hyu
2022-12-01T10:49:11.9519469Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwpoi8hyu/_remote_module_non_scriptable.py
2022-12-01T10:49:11.9519832Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:49:11.9520181Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:49:11.9520524Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:49:11.9520852Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:49:11.9521138Z skip: Need at least 2 CUDA devices (1.855s)
2022-12-01T10:49:11.9521276Z 
2022-12-01T10:49:11.9521483Z ----------------------------------------------------------------------
2022-12-01T10:49:11.9521739Z Ran 1 test in 1.855s
2022-12-01T10:49:11.9521838Z 
2022-12-01T10:49:11.9521910Z OK (skipped=1)
2022-12-01T10:49:11.9522016Z 
2022-12-01T10:49:11.9522105Z Generating XML reports...
2022-12-01T10:49:11.9522587Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaRpcTest-20221201104346.xml
2022-12-01T10:49:11.9523126Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:11.9523455Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:11.9523877Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:11.9524220Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:11.9524548Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfjbx6uvo
2022-12-01T10:49:11.9524947Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfjbx6uvo/_remote_module_non_scriptable.py
2022-12-01T10:49:11.9525430Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T10:49:11.9525650Z 
2022-12-01T10:49:11.9525723Z Running tests...
2022-12-01T10:49:11.9526008Z ----------------------------------------------------------------------
2022-12-01T10:49:11.9526410Z   test_basic_gloo_ckpt_always (__main__.TensorPipePipeWithDDPTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7302
2022-12-01T10:49:11.9526816Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7303
2022-12-01T10:49:11.9527254Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:11.9527581Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:11.9528007Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:11.9528351Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:11.9528680Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp23rv_0ba
2022-12-01T10:49:11.9529079Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp23rv_0ba/_remote_module_non_scriptable.py
2022-12-01T10:49:11.9529562Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:11.9529876Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:11.9530299Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:11.9530644Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:11.9530989Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9j9j86pj
2022-12-01T10:49:11.9531375Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9j9j86pj/_remote_module_non_scriptable.py
2022-12-01T10:49:11.9531788Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:49:11.9532165Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:49:11.9532551Z skip: Need at least 4 CUDA devices (1.744s)
2022-12-01T10:49:11.9532673Z 
2022-12-01T10:49:11.9532875Z ----------------------------------------------------------------------
2022-12-01T10:49:11.9533118Z Ran 1 test in 1.744s
2022-12-01T10:49:11.9533236Z 
2022-12-01T10:49:11.9533310Z OK (skipped=1)
2022-12-01T10:49:11.9533417Z 
2022-12-01T10:49:11.9533488Z Generating XML reports...
2022-12-01T10:49:11.9533984Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20221201104349.xml
2022-12-01T10:49:11.9534552Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:11.9534878Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:11.9535293Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:11.9535637Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:11.9535978Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbeaaqoyi
2022-12-01T10:49:11.9536362Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbeaaqoyi/_remote_module_non_scriptable.py
2022-12-01T10:49:11.9536842Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T10:49:11.9537060Z 
2022-12-01T10:49:11.9537131Z Running tests...
2022-12-01T10:49:11.9537428Z ----------------------------------------------------------------------
2022-12-01T10:49:11.9537824Z   test_basic_gloo_ckpt_except_last (__main__.TensorPipePipeWithDDPTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7335
2022-12-01T10:49:11.9538232Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7336
2022-12-01T10:49:11.9538681Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:11.9539006Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:11.9539419Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:11.9539760Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:11.9540100Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7hcg9mok
2022-12-01T10:49:11.9540661Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7hcg9mok/_remote_module_non_scriptable.py
2022-12-01T10:49:11.9541156Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:11.9541489Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:11.9541918Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:11.9542250Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:11.9542594Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsczir_xo
2022-12-01T10:49:11.9542994Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsczir_xo/_remote_module_non_scriptable.py
2022-12-01T10:49:11.9543355Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:49:11.9543699Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:49:11.9543983Z skip: Need at least 4 CUDA devices (1.744s)
2022-12-01T10:49:11.9544120Z 
2022-12-01T10:49:11.9544317Z ----------------------------------------------------------------------
2022-12-01T10:49:11.9544541Z Ran 1 test in 1.744s
2022-12-01T10:49:11.9544718Z 
2022-12-01T10:49:11.9544793Z OK (skipped=1)
2022-12-01T10:49:11.9544940Z 
2022-12-01T10:49:11.9545026Z Generating XML reports...
2022-12-01T10:49:11.9545512Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20221201104353.xml
2022-12-01T10:49:11.9546078Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:11.9546409Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:11.9546836Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:11.9547170Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:11.9547512Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplagi2pf5
2022-12-01T10:49:11.9547915Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplagi2pf5/_remote_module_non_scriptable.py
2022-12-01T10:49:11.9548394Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T10:49:11.9548596Z 
2022-12-01T10:49:11.9548667Z Running tests...
2022-12-01T10:49:11.9548966Z ----------------------------------------------------------------------
2022-12-01T10:49:11.9549371Z   test_basic_gloo_ckpt_never (__main__.TensorPipePipeWithDDPTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7368
2022-12-01T10:49:11.9549762Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7369
2022-12-01T10:49:11.9550210Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:11.9550539Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:11.9550964Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:11.9551294Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:11.9551638Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7q9o4zju
2022-12-01T10:49:11.9552039Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7q9o4zju/_remote_module_non_scriptable.py
2022-12-01T10:49:11.9552510Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:11.9552834Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:11.9553255Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:11.9553597Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:11.9553922Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2hycdt81
2022-12-01T10:49:11.9554322Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2hycdt81/_remote_module_non_scriptable.py
2022-12-01T10:49:11.9554700Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:49:11.9555047Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:49:11.9555316Z skip: Need at least 4 CUDA devices (1.745s)
2022-12-01T10:49:11.9555451Z 
2022-12-01T10:49:11.9555647Z ----------------------------------------------------------------------
2022-12-01T10:49:11.9555886Z Ran 1 test in 1.745s
2022-12-01T10:49:11.9555996Z 
2022-12-01T10:49:11.9556059Z OK (skipped=1)
2022-12-01T10:49:11.9556165Z 
2022-12-01T10:49:11.9556314Z Generating XML reports...
2022-12-01T10:49:11.9556812Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20221201104356.xml
2022-12-01T10:49:11.9557418Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:11.9557732Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:11.9558194Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:11.9558539Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:11.9558869Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpku_97kmq
2022-12-01T10:49:11.9559269Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpku_97kmq/_remote_module_non_scriptable.py
2022-12-01T10:49:11.9559747Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T10:49:11.9559963Z 
2022-12-01T10:49:11.9560034Z Running tests...
2022-12-01T10:49:11.9562487Z ----------------------------------------------------------------------
2022-12-01T10:49:11.9562917Z   test_basic_gloo_ckpt_never_find_unused (__main__.TensorPipePipeWithDDPTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7401
2022-12-01T10:49:11.9563342Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7402
2022-12-01T10:49:11.9563800Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:11.9564119Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:11.9564550Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:11.9564899Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:11.9565232Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp53h9qsa7
2022-12-01T10:49:11.9565636Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp53h9qsa7/_remote_module_non_scriptable.py
2022-12-01T10:49:11.9566121Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:11.9566451Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:11.9566866Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:11.9567208Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:11.9567551Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpha6mv107
2022-12-01T10:49:11.9567935Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpha6mv107/_remote_module_non_scriptable.py
2022-12-01T10:49:11.9568309Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:49:11.9568657Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:49:11.9568942Z skip: Need at least 4 CUDA devices (1.747s)
2022-12-01T10:49:11.9569080Z 
2022-12-01T10:49:11.9569268Z ----------------------------------------------------------------------
2022-12-01T10:49:11.9569511Z Ran 1 test in 1.748s
2022-12-01T10:49:11.9569621Z 
2022-12-01T10:49:11.9569692Z OK (skipped=1)
2022-12-01T10:49:11.9569798Z 
2022-12-01T10:49:11.9569881Z Generating XML reports...
2022-12-01T10:49:11.9570367Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20221201104359.xml
2022-12-01T10:49:11.9570933Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:11.9571262Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:11.9571674Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:11.9572017Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:11.9572513Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp78_ekmwd
2022-12-01T10:49:11.9572924Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp78_ekmwd/_remote_module_non_scriptable.py
2022-12-01T10:49:11.9573432Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T10:49:11.9573652Z 
2022-12-01T10:49:11.9573726Z Running tests...
2022-12-01T10:49:11.9574029Z ----------------------------------------------------------------------
2022-12-01T10:49:11.9574399Z   test_basic_nccl_ckpt_always (__main__.TensorPipePipeWithDDPTest) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2022-12-01T10:49:11.9574626Z 
2022-12-01T10:49:11.9574821Z ----------------------------------------------------------------------
2022-12-01T10:49:11.9575055Z Ran 1 test in 0.001s
2022-12-01T10:49:11.9575168Z 
2022-12-01T10:49:11.9575239Z OK (skipped=1)
2022-12-01T10:49:11.9575332Z 
2022-12-01T10:49:11.9575418Z Generating XML reports...
2022-12-01T10:49:11.9575918Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20221201104402.xml
2022-12-01T10:49:11.9576490Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:11.9576824Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:11.9577243Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:11.9577585Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:11.9577925Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnvv2uee3
2022-12-01T10:49:11.9578315Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnvv2uee3/_remote_module_non_scriptable.py
2022-12-01T10:49:11.9578795Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T10:49:11.9579010Z 
2022-12-01T10:49:11.9579084Z Running tests...
2022-12-01T10:49:11.9579382Z ----------------------------------------------------------------------
2022-12-01T10:49:11.9579751Z   test_basic_nccl_ckpt_except_last (__main__.TensorPipePipeWithDDPTest) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2022-12-01T10:49:11.9579981Z 
2022-12-01T10:49:11.9580175Z ----------------------------------------------------------------------
2022-12-01T10:49:11.9580565Z Ran 1 test in 0.001s
2022-12-01T10:49:11.9580681Z 
2022-12-01T10:49:11.9580741Z OK (skipped=1)
2022-12-01T10:49:11.9580850Z 
2022-12-01T10:49:11.9580936Z Generating XML reports...
2022-12-01T10:49:11.9581437Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20221201104404.xml
2022-12-01T10:49:11.9582005Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:11.9582327Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:11.9582752Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:11.9583102Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:11.9583451Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnyx0_oyc
2022-12-01T10:49:11.9583837Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnyx0_oyc/_remote_module_non_scriptable.py
2022-12-01T10:49:11.9584318Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T10:49:11.9584535Z 
2022-12-01T10:49:11.9584607Z Running tests...
2022-12-01T10:49:11.9584894Z ----------------------------------------------------------------------
2022-12-01T10:49:11.9585267Z   test_basic_nccl_ckpt_never (__main__.TensorPipePipeWithDDPTest) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2022-12-01T10:49:11.9585491Z 
2022-12-01T10:49:11.9585745Z ----------------------------------------------------------------------
2022-12-01T10:49:11.9586024Z Ran 1 test in 0.001s
2022-12-01T10:49:11.9586123Z 
2022-12-01T10:49:11.9586195Z OK (skipped=1)
2022-12-01T10:49:11.9586303Z 
2022-12-01T10:49:11.9586391Z Generating XML reports...
2022-12-01T10:49:11.9586915Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20221201104405.xml
2022-12-01T10:49:11.9587467Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:11.9587802Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:11.9588233Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:11.9588582Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:11.9588916Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgs0d0ivj
2022-12-01T10:49:11.9589325Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgs0d0ivj/_remote_module_non_scriptable.py
2022-12-01T10:49:11.9589805Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T10:49:11.9590024Z 
2022-12-01T10:49:11.9590097Z Running tests...
2022-12-01T10:49:11.9590386Z ----------------------------------------------------------------------
2022-12-01T10:49:11.9590768Z   test_basic_nccl_ckpt_never_find_unused (__main__.TensorPipePipeWithDDPTest) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2022-12-01T10:49:11.9590998Z 
2022-12-01T10:49:11.9638770Z ----------------------------------------------------------------------
2022-12-01T10:49:11.9639137Z Ran 1 test in 0.001s
2022-12-01T10:49:11.9639260Z 
2022-12-01T10:49:11.9639339Z OK (skipped=1)
2022-12-01T10:49:11.9639447Z 
2022-12-01T10:49:11.9639522Z Generating XML reports...
2022-12-01T10:49:11.9640136Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20221201104407.xml
2022-12-01T10:49:11.9640721Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:11.9641057Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:11.9641477Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:11.9641825Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:11.9642169Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph5v0ywu2
2022-12-01T10:49:11.9642559Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph5v0ywu2/_remote_module_non_scriptable.py
2022-12-01T10:49:11.9643046Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T10:49:11.9643264Z 
2022-12-01T10:49:11.9643336Z Running tests...
2022-12-01T10:49:11.9643641Z ----------------------------------------------------------------------
2022-12-01T10:49:11.9644075Z   test_async_execution_nested_with_cuda_future (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7474
2022-12-01T10:49:11.9644517Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7475
2022-12-01T10:49:11.9644845Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7476
2022-12-01T10:49:11.9645170Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 7477
2022-12-01T10:49:11.9645608Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:11.9645944Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:11.9646518Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:11.9646919Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:11.9647267Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkr5mv6ew
2022-12-01T10:49:11.9647670Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkr5mv6ew/_remote_module_non_scriptable.py
2022-12-01T10:49:11.9648160Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:11.9648478Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:11.9648904Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:11.9649248Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:11.9649580Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5ifmepsj
2022-12-01T10:49:11.9649978Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5ifmepsj/_remote_module_non_scriptable.py
2022-12-01T10:49:11.9650470Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:11.9650800Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:11.9651219Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:11.9651565Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:11.9651908Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8g3tl_ex
2022-12-01T10:49:11.9652377Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8g3tl_ex/_remote_module_non_scriptable.py
2022-12-01T10:49:11.9652859Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:11.9653189Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:11.9653618Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:11.9653952Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:11.9654293Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjl_2x09s
2022-12-01T10:49:11.9654695Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjl_2x09s/_remote_module_non_scriptable.py
2022-12-01T10:49:11.9655066Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:49:11.9655401Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:49:11.9655735Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:49:11.9656085Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:49:11.9656358Z skip: Need at least 1 CUDA device (1.958s)
2022-12-01T10:49:11.9656495Z 
2022-12-01T10:49:11.9656694Z ----------------------------------------------------------------------
2022-12-01T10:49:11.9656933Z Ran 1 test in 1.958s
2022-12-01T10:49:11.9657047Z 
2022-12-01T10:49:11.9657118Z OK (skipped=1)
2022-12-01T10:49:11.9657213Z 
2022-12-01T10:49:11.9657299Z Generating XML reports...
2022-12-01T10:49:11.9657834Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201104408.xml
2022-12-01T10:49:11.9658433Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:11.9658764Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:11.9659222Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:11.9659570Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:11.9659949Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpti6tcl02
2022-12-01T10:49:11.9660481Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpti6tcl02/_remote_module_non_scriptable.py
2022-12-01T10:49:11.9660998Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T10:49:11.9661216Z 
2022-12-01T10:49:11.9661290Z Running tests...
2022-12-01T10:49:11.9661590Z ----------------------------------------------------------------------
2022-12-01T10:49:11.9662018Z   test_async_execution_with_cuda_future (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7529
2022-12-01T10:49:11.9662459Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7530
2022-12-01T10:49:11.9662790Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7531
2022-12-01T10:49:11.9663107Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 7532
2022-12-01T10:49:11.9663558Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:11.9663884Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:11.9664317Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:11.9664648Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:11.9664993Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8z0_8i7w
2022-12-01T10:49:11.9665394Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8z0_8i7w/_remote_module_non_scriptable.py
2022-12-01T10:49:11.9665879Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:11.9666199Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:11.9666626Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:11.9666969Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:11.9667297Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7756uc5l
2022-12-01T10:49:11.9667696Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7756uc5l/_remote_module_non_scriptable.py
2022-12-01T10:49:11.9668179Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:11.9668505Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:11.9668920Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:11.9669263Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:11.9669603Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4_eicne_
2022-12-01T10:49:11.9669988Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4_eicne_/_remote_module_non_scriptable.py
2022-12-01T10:49:11.9670470Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:11.9670799Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:11.9671226Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:11.9671561Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:11.9671901Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdsz2o1tb
2022-12-01T10:49:11.9672358Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdsz2o1tb/_remote_module_non_scriptable.py
2022-12-01T10:49:11.9672770Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:49:11.9673103Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:49:11.9673445Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:49:11.9673789Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:49:11.9674058Z skip: Need at least 1 CUDA device (1.954s)
2022-12-01T10:49:11.9674192Z 
2022-12-01T10:49:11.9674389Z ----------------------------------------------------------------------
2022-12-01T10:49:11.9674630Z Ran 1 test in 1.954s
2022-12-01T10:49:11.9674746Z 
2022-12-01T10:49:11.9674817Z OK (skipped=1)
2022-12-01T10:49:11.9674913Z 
2022-12-01T10:49:11.9674998Z Generating XML reports...
2022-12-01T10:49:11.9675536Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201104411.xml
2022-12-01T10:49:11.9676134Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:11.9676451Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:11.9676880Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:11.9677224Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:11.9677563Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpywd5syyq
2022-12-01T10:49:11.9677949Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpywd5syyq/_remote_module_non_scriptable.py
2022-12-01T10:49:11.9678428Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T10:49:11.9678645Z 
2022-12-01T10:49:11.9678719Z Running tests...
2022-12-01T10:49:11.9679006Z ----------------------------------------------------------------------
2022-12-01T10:49:11.9679457Z   test_cuda_future_callback_changes_devices (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7584
2022-12-01T10:49:11.9679898Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7585
2022-12-01T10:49:11.9680224Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7586
2022-12-01T10:49:11.9680537Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 7587
2022-12-01T10:49:11.9680982Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:11.9681309Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:11.9681735Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:11.9682071Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:11.9682413Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqsd4iaal
2022-12-01T10:49:11.9682889Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:11.9683204Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:11.9683631Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:11.9683974Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:11.9684314Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk9ca_5vd
2022-12-01T10:49:11.9684698Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqsd4iaal/_remote_module_non_scriptable.py
2022-12-01T10:49:11.9685137Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk9ca_5vd/_remote_module_non_scriptable.py
2022-12-01T10:49:11.9685652Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:11.9685990Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:11.9686403Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:11.9686748Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:11.9687088Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw8elcobx
2022-12-01T10:49:11.9687476Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw8elcobx/_remote_module_non_scriptable.py
2022-12-01T10:49:11.9687961Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:11.9688292Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:11.9688721Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:11.9689052Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:11.9689397Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvwgpj0b0
2022-12-01T10:49:11.9689794Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvwgpj0b0/_remote_module_non_scriptable.py
2022-12-01T10:49:11.9690156Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:49:11.9690503Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:49:11.9690846Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:49:11.9691187Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:49:11.9691463Z skip: Need at least 2 CUDA devices (1.954s)
2022-12-01T10:49:11.9691601Z 
2022-12-01T10:49:11.9691800Z ----------------------------------------------------------------------
2022-12-01T10:49:11.9692039Z Ran 1 test in 1.955s
2022-12-01T10:49:11.9692152Z 
2022-12-01T10:49:11.9692210Z OK (skipped=1)
2022-12-01T10:49:11.9692431Z 
2022-12-01T10:49:11.9692518Z Generating XML reports...
2022-12-01T10:49:11.9693057Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201104415.xml
2022-12-01T10:49:11.9693656Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:11.9693972Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:11.9694399Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:11.9694744Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:11.9695091Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphj8y4qje
2022-12-01T10:49:11.9695478Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphj8y4qje/_remote_module_non_scriptable.py
2022-12-01T10:49:11.9695960Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T10:49:11.9696178Z 
2022-12-01T10:49:11.9696249Z Running tests...
2022-12-01T10:49:11.9696536Z ----------------------------------------------------------------------
2022-12-01T10:49:11.9696991Z   test_cuda_future_can_extract_cuda_sparse_tensor (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7639
2022-12-01T10:49:11.9697435Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7640
2022-12-01T10:49:11.9697835Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7641
2022-12-01T10:49:11.9698150Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 7642
2022-12-01T10:49:11.9698630Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:11.9698957Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:11.9699384Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:11.9699717Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:11.9700059Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_ab68b1x
2022-12-01T10:49:11.9700594Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_ab68b1x/_remote_module_non_scriptable.py
2022-12-01T10:49:11.9701071Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:11.9701402Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:11.9701832Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:11.9702174Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:11.9702505Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb1hqwwro
2022-12-01T10:49:11.9702904Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb1hqwwro/_remote_module_non_scriptable.py
2022-12-01T10:49:11.9703392Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:11.9703704Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:11.9704131Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:11.9704478Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:11.9704821Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp30r3yjta
2022-12-01T10:49:11.9705204Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp30r3yjta/_remote_module_non_scriptable.py
2022-12-01T10:49:11.9705687Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:11.9706016Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:11.9706446Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:11.9706776Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:11.9707116Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfg6jo2as
2022-12-01T10:49:11.9707516Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfg6jo2as/_remote_module_non_scriptable.py
2022-12-01T10:49:11.9707882Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:49:11.9708235Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:49:11.9708577Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:49:11.9708922Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:49:11.9709194Z skip: Need at least 1 CUDA device (1.853s)
2022-12-01T10:49:11.9709333Z 
2022-12-01T10:49:11.9709532Z ----------------------------------------------------------------------
2022-12-01T10:49:11.9709771Z Ran 1 test in 1.853s
2022-12-01T10:49:11.9709884Z 
2022-12-01T10:49:11.9709943Z OK (skipped=1)
2022-12-01T10:49:11.9710050Z 
2022-12-01T10:49:11.9710134Z Generating XML reports...
2022-12-01T10:49:11.9710728Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201104418.xml
2022-12-01T10:49:11.9711352Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:11.9711667Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:11.9712095Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:11.9712441Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:11.9712784Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwuyr75cf
2022-12-01T10:49:11.9713172Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwuyr75cf/_remote_module_non_scriptable.py
2022-12-01T10:49:11.9713652Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T10:49:11.9713869Z 
2022-12-01T10:49:11.9713942Z Running tests...
2022-12-01T10:49:11.9714229Z ----------------------------------------------------------------------
2022-12-01T10:49:11.9714676Z   test_cuda_future_can_extract_cuda_tensor (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7694
2022-12-01T10:49:11.9715122Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7695
2022-12-01T10:49:11.9715448Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7696
2022-12-01T10:49:11.9715761Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 7697
2022-12-01T10:49:11.9716206Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:11.9716534Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:11.9716947Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:11.9717291Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:11.9717635Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3jrrmhwc
2022-12-01T10:49:11.9718033Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3jrrmhwc/_remote_module_non_scriptable.py
2022-12-01T10:49:11.9718503Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:11.9718829Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:11.9719254Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:11.9719596Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:11.9719921Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_0e98dyw
2022-12-01T10:49:11.9720400Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:11.9720730Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:11.9721142Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:11.9721487Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:11.9721835Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk26emm1q
2022-12-01T10:49:11.9722232Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_0e98dyw/_remote_module_non_scriptable.py
2022-12-01T10:49:11.9722619Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk26emm1q/_remote_module_non_scriptable.py
2022-12-01T10:49:11.9723102Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:11.9723428Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:11.9723873Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:11.9724249Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:11.9724651Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfv4asvy9
2022-12-01T10:49:11.9725045Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfv4asvy9/_remote_module_non_scriptable.py
2022-12-01T10:49:11.9725409Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:49:11.9725756Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:49:11.9726100Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:49:11.9726443Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:49:11.9726715Z skip: Need at least 1 CUDA device (1.855s)
2022-12-01T10:49:11.9726850Z 
2022-12-01T10:49:11.9727050Z ----------------------------------------------------------------------
2022-12-01T10:49:11.9727288Z Ran 1 test in 1.855s
2022-12-01T10:49:11.9727400Z 
2022-12-01T10:49:11.9727459Z OK (skipped=1)
2022-12-01T10:49:11.9727566Z 
2022-12-01T10:49:11.9727650Z Generating XML reports...
2022-12-01T10:49:11.9728183Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201104422.xml
2022-12-01T10:49:11.9728779Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:11.9729094Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:11.9729520Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:11.9729865Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:11.9730197Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe6ft9gac
2022-12-01T10:49:11.9730597Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe6ft9gac/_remote_module_non_scriptable.py
2022-12-01T10:49:11.9731079Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T10:49:11.9731293Z 
2022-12-01T10:49:11.9731365Z Running tests...
2022-12-01T10:49:11.9731653Z ----------------------------------------------------------------------
2022-12-01T10:49:11.9732119Z   test_cuda_future_can_extract_custom_class_with_cuda_sparse_tensor (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7749
2022-12-01T10:49:11.9732667Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7750
2022-12-01T10:49:11.9732998Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7751
2022-12-01T10:49:11.9733311Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 7752
2022-12-01T10:49:11.9733761Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:11.9734090Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:11.9734505Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:11.9734842Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:11.9735182Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpec159kys
2022-12-01T10:49:11.9735576Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpec159kys/_remote_module_non_scriptable.py
2022-12-01T10:49:11.9736044Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:11.9736405Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:11.9736873Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:11.9737215Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:11.9737544Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkb3bo4nd
2022-12-01T10:49:11.9737943Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkb3bo4nd/_remote_module_non_scriptable.py
2022-12-01T10:49:11.9738426Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:11.9738740Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:11.9739168Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:11.9739514Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:11.9739855Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbf3fbv0f
2022-12-01T10:49:11.9740242Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbf3fbv0f/_remote_module_non_scriptable.py
2022-12-01T10:49:11.9740910Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:11.9741240Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:11.9741655Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:11.9742003Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:11.9742345Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpet9rzn3f
2022-12-01T10:49:11.9742746Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpet9rzn3f/_remote_module_non_scriptable.py
2022-12-01T10:49:11.9743112Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:49:11.9743464Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:49:11.9743808Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:49:11.9744152Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:49:11.9744422Z skip: Need at least 1 CUDA device (1.853s)
2022-12-01T10:49:11.9744558Z 
2022-12-01T10:49:11.9744760Z ----------------------------------------------------------------------
2022-12-01T10:49:11.9744997Z Ran 1 test in 1.853s
2022-12-01T10:49:11.9745109Z 
2022-12-01T10:49:11.9745169Z OK (skipped=1)
2022-12-01T10:49:11.9745276Z 
2022-12-01T10:49:11.9745360Z Generating XML reports...
2022-12-01T10:49:11.9745901Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201104425.xml
2022-12-01T10:49:11.9746495Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:11.9746815Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:11.9747239Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:11.9747583Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:11.9747914Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnu3rpu97
2022-12-01T10:49:11.9748312Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnu3rpu97/_remote_module_non_scriptable.py
2022-12-01T10:49:11.9748791Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T10:49:11.9749010Z 
2022-12-01T10:49:11.9749082Z Running tests...
2022-12-01T10:49:11.9749427Z ----------------------------------------------------------------------
2022-12-01T10:49:11.9749929Z   test_cuda_future_can_extract_custom_class_with_cuda_tensor (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7804
2022-12-01T10:49:11.9750384Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7805
2022-12-01T10:49:11.9750715Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7806
2022-12-01T10:49:11.9751031Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 7807
2022-12-01T10:49:11.9751480Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:11.9751810Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:11.9752225Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:11.9752572Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:11.9752919Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5yl7psgw
2022-12-01T10:49:11.9753321Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5yl7psgw/_remote_module_non_scriptable.py
2022-12-01T10:49:11.9753796Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:11.9754123Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:11.9754547Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:11.9754882Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:11.9755225Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgtt8ihdq
2022-12-01T10:49:11.9755630Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgtt8ihdq/_remote_module_non_scriptable.py
2022-12-01T10:49:11.9756116Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:11.9756434Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:11.9756857Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:11.9757195Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:11.9757534Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1w507har
2022-12-01T10:49:11.9757919Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1w507har/_remote_module_non_scriptable.py
2022-12-01T10:49:11.9758403Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:11.9758728Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:11.9759148Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:11.9759492Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:11.9759836Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2zmvcmw8
2022-12-01T10:49:11.9760230Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2zmvcmw8/_remote_module_non_scriptable.py
2022-12-01T10:49:11.9760590Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:49:11.9760939Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:49:11.9761281Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:49:11.9761614Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:49:11.9761930Z skip: Need at least 1 CUDA device (1.854s)
2022-12-01T10:49:11.9762067Z 
2022-12-01T10:49:11.9762296Z ----------------------------------------------------------------------
2022-12-01T10:49:11.9762534Z Ran 1 test in 1.854s
2022-12-01T10:49:11.9762634Z 
2022-12-01T10:49:11.9762706Z OK (skipped=1)
2022-12-01T10:49:11.9762815Z 
2022-12-01T10:49:11.9762902Z Generating XML reports...
2022-12-01T10:49:11.9763437Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201104429.xml
2022-12-01T10:49:11.9764021Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:11.9764348Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:11.9764774Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:11.9765116Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:11.9765449Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwplrtf2a
2022-12-01T10:49:11.9765854Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwplrtf2a/_remote_module_non_scriptable.py
2022-12-01T10:49:11.9766334Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T10:49:11.9766551Z 
2022-12-01T10:49:11.9766623Z Running tests...
2022-12-01T10:49:11.9766909Z ----------------------------------------------------------------------
2022-12-01T10:49:11.9767369Z   test_cuda_future_can_extract_list_with_cuda_sparse_tensor (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7859
2022-12-01T10:49:11.9767821Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7860
2022-12-01T10:49:11.9768138Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7861
2022-12-01T10:49:11.9768468Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 7862
2022-12-01T10:49:11.9768912Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:11.9769243Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:11.9769657Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:11.9770003Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:11.9770344Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp6aayq4r
2022-12-01T10:49:11.9770748Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp6aayq4r/_remote_module_non_scriptable.py
2022-12-01T10:49:11.9771223Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:11.9771555Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:11.9771981Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:11.9772394Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:11.9772738Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpksgt068e
2022-12-01T10:49:11.9773216Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:11.9773544Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:11.9773960Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:11.9774302Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:11.9774643Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphstvo73d
2022-12-01T10:49:11.9775077Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpksgt068e/_remote_module_non_scriptable.py
2022-12-01T10:49:11.9775514Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphstvo73d/_remote_module_non_scriptable.py
2022-12-01T10:49:11.9775998Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:11.9776323Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:11.9776734Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:11.9777077Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:11.9777421Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprw612wso
2022-12-01T10:49:11.9777858Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprw612wso/_remote_module_non_scriptable.py
2022-12-01T10:49:11.9778222Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:49:11.9778570Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:49:11.9778915Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:49:11.9779249Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:49:11.9779527Z skip: Need at least 1 CUDA device (1.854s)
2022-12-01T10:49:11.9779662Z 
2022-12-01T10:49:11.9779860Z ----------------------------------------------------------------------
2022-12-01T10:49:11.9780100Z Ran 1 test in 1.854s
2022-12-01T10:49:11.9780200Z 
2022-12-01T10:49:11.9780271Z OK (skipped=1)
2022-12-01T10:49:11.9780505Z 
2022-12-01T10:49:11.9780589Z Generating XML reports...
2022-12-01T10:49:11.9784407Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201104432.xml
2022-12-01T10:49:11.9785810Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:11.9786529Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:11.9787476Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:11.9788226Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:11.9788948Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp04lq_o7q
2022-12-01T10:49:11.9789803Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp04lq_o7q/_remote_module_non_scriptable.py
2022-12-01T10:49:11.9790837Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T10:49:11.9791308Z 
2022-12-01T10:49:11.9791464Z Running tests...
2022-12-01T10:49:11.9792083Z ----------------------------------------------------------------------
2022-12-01T10:49:11.9793045Z   test_cuda_future_can_extract_list_with_cuda_tensor (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7914
2022-12-01T10:49:11.9794014Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7915
2022-12-01T10:49:11.9794696Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7916
2022-12-01T10:49:11.9795391Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 7917
2022-12-01T10:49:11.9796370Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:11.9797062Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:11.9797972Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:11.9798843Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:11.9799640Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2hxn4op3
2022-12-01T10:49:11.9800480Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2hxn4op3/_remote_module_non_scriptable.py
2022-12-01T10:49:11.9801524Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:11.9801783Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:11.9802387Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:11.9802666Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:11.9803062Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp690nl4q7
2022-12-01T10:49:11.9803477Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp690nl4q7/_remote_module_non_scriptable.py
2022-12-01T10:49:11.9804064Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:11.9804326Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:11.9804936Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:11.9805219Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:11.9805611Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp65_s3xnm
2022-12-01T10:49:11.9806205Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:11.9806465Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:11.9807073Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:11.9807363Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:11.9807744Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwdgek__b
2022-12-01T10:49:11.9808155Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp65_s3xnm/_remote_module_non_scriptable.py
2022-12-01T10:49:11.9808569Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwdgek__b/_remote_module_non_scriptable.py
2022-12-01T10:49:11.9808919Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:49:11.9809271Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:49:11.9809625Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:49:11.9809977Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:49:11.9810184Z skip: Need at least 1 CUDA device (1.853s)
2022-12-01T10:49:11.9810199Z 
2022-12-01T10:49:11.9810619Z ----------------------------------------------------------------------
2022-12-01T10:49:11.9810757Z Ran 1 test in 1.853s
2022-12-01T10:49:11.9810768Z 
2022-12-01T10:49:11.9810921Z OK (skipped=1)
2022-12-01T10:49:11.9810930Z 
2022-12-01T10:49:11.9811108Z Generating XML reports...
2022-12-01T10:49:11.9812024Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201104435.xml
2022-12-01T10:49:11.9812695Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:11.9812960Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:11.9813580Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:11.9813870Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:11.9814316Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmkxk8ks_
2022-12-01T10:49:11.9814792Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmkxk8ks_/_remote_module_non_scriptable.py
2022-12-01T10:49:11.9815379Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T10:49:11.9815391Z 
2022-12-01T10:49:11.9815536Z Running tests...
2022-12-01T10:49:11.9815962Z ----------------------------------------------------------------------
2022-12-01T10:49:11.9816555Z   test_cuda_future_device_as_device (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7969
2022-12-01T10:49:11.9816892Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7970
2022-12-01T10:49:11.9817218Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7971
2022-12-01T10:49:11.9817552Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 7972
2022-12-01T10:49:11.9818140Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:11.9818408Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:11.9819030Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:11.9819323Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:11.9819729Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzewhaz7e
2022-12-01T10:49:11.9820157Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzewhaz7e/_remote_module_non_scriptable.py
2022-12-01T10:49:11.9887821Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:11.9888096Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:11.9888726Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:11.9888998Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:11.9889397Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_eonx55u
2022-12-01T10:49:11.9890002Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:11.9890265Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:11.9890892Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:11.9891184Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:11.9891586Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzfl_sjqi
2022-12-01T10:49:11.9892014Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_eonx55u/_remote_module_non_scriptable.py
2022-12-01T10:49:11.9892466Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzfl_sjqi/_remote_module_non_scriptable.py
2022-12-01T10:49:11.9893066Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:11.9893328Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:11.9893954Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:11.9894244Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:11.9894645Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmchpp9tz
2022-12-01T10:49:11.9895077Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmchpp9tz/_remote_module_non_scriptable.py
2022-12-01T10:49:11.9895546Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:49:11.9895911Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:49:11.9896302Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:49:11.9896654Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:49:11.9896860Z skip: Need at least 1 CUDA device (1.853s)
2022-12-01T10:49:11.9896873Z 
2022-12-01T10:49:11.9897307Z ----------------------------------------------------------------------
2022-12-01T10:49:11.9897472Z Ran 1 test in 1.853s
2022-12-01T10:49:11.9897482Z 
2022-12-01T10:49:11.9897638Z OK (skipped=1)
2022-12-01T10:49:11.9897647Z 
2022-12-01T10:49:11.9897834Z Generating XML reports...
2022-12-01T10:49:11.9898784Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201104439.xml
2022-12-01T10:49:11.9899382Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:11.9899650Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:11.9900266Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:11.9900799Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:11.9901198Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp87pn7fux
2022-12-01T10:49:11.9901617Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp87pn7fux/_remote_module_non_scriptable.py
2022-12-01T10:49:11.9902212Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T10:49:11.9902224Z 
2022-12-01T10:49:11.9902376Z Running tests...
2022-12-01T10:49:11.9902803Z ----------------------------------------------------------------------
2022-12-01T10:49:11.9903372Z   test_cuda_future_device_as_int (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8024
2022-12-01T10:49:11.9903715Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8025
2022-12-01T10:49:11.9904046Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8026
2022-12-01T10:49:11.9904370Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8027
2022-12-01T10:49:11.9904975Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:11.9905235Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:11.9905862Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:11.9906151Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:11.9906550Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb7mdxg98
2022-12-01T10:49:11.9906953Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb7mdxg98/_remote_module_non_scriptable.py
2022-12-01T10:49:11.9907554Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:11.9907812Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:11.9908431Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:11.9908715Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:11.9909121Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp3u23hrh
2022-12-01T10:49:11.9909542Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp3u23hrh/_remote_module_non_scriptable.py
2022-12-01T10:49:11.9910251Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:11.9910558Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:11.9911180Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:11.9911472Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:11.9911867Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp0a21e4f
2022-12-01T10:49:11.9912291Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp0a21e4f/_remote_module_non_scriptable.py
2022-12-01T10:49:11.9912891Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:11.9913160Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:11.9913797Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:11.9914084Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:11.9914468Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptxl5n4l5
2022-12-01T10:49:11.9914890Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptxl5n4l5/_remote_module_non_scriptable.py
2022-12-01T10:49:11.9915252Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:49:11.9915608Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:49:11.9915959Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:49:11.9916309Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:49:11.9916521Z skip: Need at least 1 CUDA device (1.853s)
2022-12-01T10:49:11.9916530Z 
2022-12-01T10:49:11.9916953Z ----------------------------------------------------------------------
2022-12-01T10:49:11.9917094Z Ran 1 test in 1.853s
2022-12-01T10:49:11.9917121Z 
2022-12-01T10:49:11.9917254Z OK (skipped=1)
2022-12-01T10:49:11.9917263Z 
2022-12-01T10:49:11.9917439Z Generating XML reports...
2022-12-01T10:49:11.9918352Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201104442.xml
2022-12-01T10:49:11.9918947Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:11.9919209Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:11.9919836Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:11.9920122Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:11.9920527Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9h97lnpd
2022-12-01T10:49:11.9920932Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9h97lnpd/_remote_module_non_scriptable.py
2022-12-01T10:49:11.9921522Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T10:49:11.9921533Z 
2022-12-01T10:49:11.9921685Z Running tests...
2022-12-01T10:49:11.9922104Z ----------------------------------------------------------------------
2022-12-01T10:49:11.9922687Z   test_cuda_future_device_as_str (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8079
2022-12-01T10:49:11.9923015Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8080
2022-12-01T10:49:11.9923344Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8081
2022-12-01T10:49:11.9923671Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8082
2022-12-01T10:49:11.9924321Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:11.9924609Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:11.9925241Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:11.9925524Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:11.9925924Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqj8dau3x
2022-12-01T10:49:11.9926347Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqj8dau3x/_remote_module_non_scriptable.py
2022-12-01T10:49:11.9926952Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:11.9927216Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:11.9927840Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:11.9928112Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:11.9928515Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo62hbdaf
2022-12-01T10:49:11.9928945Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo62hbdaf/_remote_module_non_scriptable.py
2022-12-01T10:49:11.9929548Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:11.9929808Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:11.9930428Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:11.9930715Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:11.9931114Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpueatfbox
2022-12-01T10:49:11.9931538Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpueatfbox/_remote_module_non_scriptable.py
2022-12-01T10:49:11.9932131Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:11.9932465Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:11.9933081Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:11.9933372Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:11.9933766Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_9_ibpn3
2022-12-01T10:49:11.9934180Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_9_ibpn3/_remote_module_non_scriptable.py
2022-12-01T10:49:11.9934535Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:49:11.9934890Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:49:11.9935231Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:49:11.9935573Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:49:11.9935782Z skip: Need at least 1 CUDA device (1.853s)
2022-12-01T10:49:11.9935791Z 
2022-12-01T10:49:11.9936208Z ----------------------------------------------------------------------
2022-12-01T10:49:11.9936361Z Ran 1 test in 1.853s
2022-12-01T10:49:11.9936371Z 
2022-12-01T10:49:11.9936515Z OK (skipped=1)
2022-12-01T10:49:11.9936527Z 
2022-12-01T10:49:11.9936702Z Generating XML reports...
2022-12-01T10:49:11.9937633Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201104446.xml
2022-12-01T10:49:11.9938288Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:11.9938533Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:11.9939196Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:11.9939481Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:11.9939874Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb_ab7eoq
2022-12-01T10:49:11.9940296Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb_ab7eoq/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0031622Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T10:49:12.0031640Z 
2022-12-01T10:49:12.0031798Z Running tests...
2022-12-01T10:49:12.0032220Z ----------------------------------------------------------------------
2022-12-01T10:49:12.0032809Z   test_cuda_future_device_not_cuda (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8134
2022-12-01T10:49:12.0033126Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8135
2022-12-01T10:49:12.0033452Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8136
2022-12-01T10:49:12.0033778Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8137
2022-12-01T10:49:12.0034391Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0034653Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0035279Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0035564Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0035969Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp15wbwyyx
2022-12-01T10:49:12.0036366Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp15wbwyyx/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0036974Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0037235Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0037864Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0038161Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0038565Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzw8yrdxa
2022-12-01T10:49:12.0038987Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzw8yrdxa/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0039582Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0039839Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0040438Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0040717Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0041115Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8leuly7c
2022-12-01T10:49:12.0041531Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8leuly7c/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0042133Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0042396Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0043017Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0043425Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0043885Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpna2e8whf
2022-12-01T10:49:12.0044289Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpna2e8whf/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0044647Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:49:12.0045004Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:49:12.0045356Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:49:12.0045713Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:49:12.0045931Z skip: Need at least 1 CUDA device (1.851s)
2022-12-01T10:49:12.0045940Z 
2022-12-01T10:49:12.0046368Z ----------------------------------------------------------------------
2022-12-01T10:49:12.0046529Z Ran 1 test in 1.851s
2022-12-01T10:49:12.0046541Z 
2022-12-01T10:49:12.0046675Z OK (skipped=1)
2022-12-01T10:49:12.0046687Z 
2022-12-01T10:49:12.0046864Z Generating XML reports...
2022-12-01T10:49:12.0047794Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201104449.xml
2022-12-01T10:49:12.0048408Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0048671Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0049302Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0049597Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0049998Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq4eyyf89
2022-12-01T10:49:12.0050435Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq4eyyf89/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0051008Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T10:49:12.0051022Z 
2022-12-01T10:49:12.0051174Z Running tests...
2022-12-01T10:49:12.0051589Z ----------------------------------------------------------------------
2022-12-01T10:49:12.0052195Z   test_cuda_future_modify_tensor_inplace (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8189
2022-12-01T10:49:12.0052592Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8190
2022-12-01T10:49:12.0052924Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8191
2022-12-01T10:49:12.0053242Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8192
2022-12-01T10:49:12.0053856Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0054116Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0054721Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0055003Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0055401Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4lenshdl
2022-12-01T10:49:12.0055824Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4lenshdl/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0056428Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0056682Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0057298Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0057637Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0058061Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_wmmrr2n
2022-12-01T10:49:12.0058491Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_wmmrr2n/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0059098Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0059357Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0059970Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0060258Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0060747Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1v0pbe65
2022-12-01T10:49:12.0061179Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1v0pbe65/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0061790Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0062027Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0062653Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0062947Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0063348Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz8w8wj88
2022-12-01T10:49:12.0063769Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz8w8wj88/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0064124Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:49:12.0064484Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:49:12.0064837Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:49:12.0065176Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:49:12.0065398Z skip: Need at least 1 CUDA device (1.850s)
2022-12-01T10:49:12.0065408Z 
2022-12-01T10:49:12.0065821Z ----------------------------------------------------------------------
2022-12-01T10:49:12.0065979Z Ran 1 test in 1.851s
2022-12-01T10:49:12.0065989Z 
2022-12-01T10:49:12.0066138Z OK (skipped=1)
2022-12-01T10:49:12.0066147Z 
2022-12-01T10:49:12.0066330Z Generating XML reports...
2022-12-01T10:49:12.0067260Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201104452.xml
2022-12-01T10:49:12.0067869Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0068138Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0068739Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0069023Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0069418Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp62ol4ldi
2022-12-01T10:49:12.0069836Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp62ol4ldi/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0070422Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T10:49:12.0070434Z 
2022-12-01T10:49:12.0070583Z Running tests...
2022-12-01T10:49:12.0071003Z ----------------------------------------------------------------------
2022-12-01T10:49:12.0071660Z   test_cuda_future_replace_tensor (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8244
2022-12-01T10:49:12.0072068Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8245
2022-12-01T10:49:12.0072397Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8246
2022-12-01T10:49:12.0072735Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8247
2022-12-01T10:49:12.0073364Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0073631Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0074263Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0074548Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0074948Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcc8vsk0i
2022-12-01T10:49:12.0075371Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcc8vsk0i/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0075962Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0076226Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0076840Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0077122Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0077519Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgz1xewm_
2022-12-01T10:49:12.0077942Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgz1xewm_/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0078542Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0078801Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0079413Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0079685Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0080078Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp983_kahg
2022-12-01T10:49:12.0080492Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp983_kahg/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0081086Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0081345Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0081966Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0082246Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0082648Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjlfw_0gb
2022-12-01T10:49:12.0083048Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjlfw_0gb/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0083400Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:49:12.0083754Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:49:12.0084102Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:49:12.0084453Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:49:12.0084673Z skip: Need at least 1 CUDA device (1.853s)
2022-12-01T10:49:12.0084683Z 
2022-12-01T10:49:12.0085107Z ----------------------------------------------------------------------
2022-12-01T10:49:12.0085262Z Ran 1 test in 1.853s
2022-12-01T10:49:12.0085272Z 
2022-12-01T10:49:12.0085468Z OK (skipped=1)
2022-12-01T10:49:12.0085481Z 
2022-12-01T10:49:12.0085687Z Generating XML reports...
2022-12-01T10:49:12.0086629Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201104456.xml
2022-12-01T10:49:12.0087241Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0087497Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0088123Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0088415Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0088819Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjlm45rj3
2022-12-01T10:49:12.0089293Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjlm45rj3/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0089877Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T10:49:12.0089892Z 
2022-12-01T10:49:12.0090033Z Running tests...
2022-12-01T10:49:12.0090461Z ----------------------------------------------------------------------
2022-12-01T10:49:12.0091059Z   test_cuda_future_value_on_bad_device (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8299
2022-12-01T10:49:12.0091401Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8300
2022-12-01T10:49:12.0091732Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8301
2022-12-01T10:49:12.0092055Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8302
2022-12-01T10:49:12.0092723Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0092985Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0093590Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0093874Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0094282Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpms5ru5yq
2022-12-01T10:49:12.0094705Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpms5ru5yq/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0095306Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0095569Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0096177Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0096496Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0096893Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw2_hrf8r
2022-12-01T10:49:12.0097298Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw2_hrf8r/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0097905Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0098164Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0098791Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0099071Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0099469Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx6c7ol61
2022-12-01T10:49:12.0099878Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx6c7ol61/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0225604Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0225990Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0226635Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0226927Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0227326Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph8i5pe34
2022-12-01T10:49:12.0227758Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph8i5pe34/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0228118Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:49:12.0228476Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:49:12.0228828Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:49:12.0229188Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:49:12.0229384Z skip: Need at least 2 CUDA devices (1.956s)
2022-12-01T10:49:12.0229398Z 
2022-12-01T10:49:12.0229820Z ----------------------------------------------------------------------
2022-12-01T10:49:12.0229979Z Ran 1 test in 1.956s
2022-12-01T10:49:12.0229988Z 
2022-12-01T10:49:12.0230143Z OK (skipped=1)
2022-12-01T10:49:12.0230153Z 
2022-12-01T10:49:12.0230339Z Generating XML reports...
2022-12-01T10:49:12.0231267Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201104459.xml
2022-12-01T10:49:12.0231904Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0232167Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0232800Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0233074Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0233476Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp59ypo9hz
2022-12-01T10:49:12.0233901Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp59ypo9hz/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0234500Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T10:49:12.0234513Z 
2022-12-01T10:49:12.0234677Z Running tests...
2022-12-01T10:49:12.0235098Z ----------------------------------------------------------------------
2022-12-01T10:49:12.0236729Z   test_custom_stream (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/79750 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.233s)
2022-12-01T10:49:12.0236747Z 
2022-12-01T10:49:12.0237156Z ----------------------------------------------------------------------
2022-12-01T10:49:12.0237317Z Ran 1 test in 0.234s
2022-12-01T10:49:12.0237327Z 
2022-12-01T10:49:12.0237477Z OK (skipped=1)
2022-12-01T10:49:12.0237486Z 
2022-12-01T10:49:12.0237642Z Generating XML reports...
2022-12-01T10:49:12.0238565Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201104503.xml
2022-12-01T10:49:12.0239165Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0239419Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0240106Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0240466Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0240882Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp879bd8la
2022-12-01T10:49:12.0241302Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp879bd8la/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0241876Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T10:49:12.0241902Z 
2022-12-01T10:49:12.0242034Z Running tests...
2022-12-01T10:49:12.0242453Z ----------------------------------------------------------------------
2022-12-01T10:49:12.0243023Z   test_custom_stream_multi (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8364
2022-12-01T10:49:12.0243360Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8365
2022-12-01T10:49:12.0243692Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8366
2022-12-01T10:49:12.0244029Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8367
2022-12-01T10:49:12.0244632Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0244903Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0245512Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0245803Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0246207Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5qtlzvtz
2022-12-01T10:49:12.0246625Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5qtlzvtz/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0247231Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0247501Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0248134Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0248430Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0248839Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4jc0r4kk
2022-12-01T10:49:12.0249417Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0249683Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0250298Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0250589Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0250995Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1gfprgp0
2022-12-01T10:49:12.0251418Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4jc0r4kk/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0251846Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1gfprgp0/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0252506Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0252748Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0253369Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0253648Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0254043Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4qouahdd
2022-12-01T10:49:12.0254518Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4qouahdd/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0254951Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:49:12.0255305Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:49:12.0255653Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:49:12.0256000Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:49:12.0256196Z skip: Need at least 2 CUDA devices (1.954s)
2022-12-01T10:49:12.0256208Z 
2022-12-01T10:49:12.0256626Z ----------------------------------------------------------------------
2022-12-01T10:49:12.0256782Z Ran 1 test in 1.954s
2022-12-01T10:49:12.0256792Z 
2022-12-01T10:49:12.0256946Z OK (skipped=1)
2022-12-01T10:49:12.0256955Z 
2022-12-01T10:49:12.0257137Z Generating XML reports...
2022-12-01T10:49:12.0258066Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201104504.xml
2022-12-01T10:49:12.0258674Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0258933Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0259551Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0259818Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0260218Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwfg4pqw1
2022-12-01T10:49:12.0327004Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwfg4pqw1/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0327639Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T10:49:12.0327649Z 
2022-12-01T10:49:12.0327806Z Running tests...
2022-12-01T10:49:12.0328224Z ----------------------------------------------------------------------
2022-12-01T10:49:12.0328798Z   test_custom_stream_nested (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8419
2022-12-01T10:49:12.0329136Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8420
2022-12-01T10:49:12.0329446Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8421
2022-12-01T10:49:12.0329776Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8422
2022-12-01T10:49:12.0330369Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0330630Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0331260Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0331558Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0331965Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2p1766r3
2022-12-01T10:49:12.0332453Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2p1766r3/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0333054Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0333290Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0333908Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0334188Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0334587Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbdqau93n
2022-12-01T10:49:12.0335140Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbdqau93n/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0335792Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0336056Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0336668Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0336938Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0337337Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsqvmhm1m
2022-12-01T10:49:12.0337757Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsqvmhm1m/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0338352Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0338605Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0339220Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0339503Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0339907Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf2vrh690
2022-12-01T10:49:12.0340326Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf2vrh690/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0340821Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:49:12.0341176Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:49:12.0341527Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:49:12.0341891Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:49:12.0342112Z skip: Need at least 2 CUDA devices (1.854s)
2022-12-01T10:49:12.0342128Z 
2022-12-01T10:49:12.0342563Z ----------------------------------------------------------------------
2022-12-01T10:49:12.0342720Z Ran 1 test in 1.855s
2022-12-01T10:49:12.0342731Z 
2022-12-01T10:49:12.0342882Z OK (skipped=1)
2022-12-01T10:49:12.0342891Z 
2022-12-01T10:49:12.0343049Z Generating XML reports...
2022-12-01T10:49:12.0343992Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201104508.xml
2022-12-01T10:49:12.0344606Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0344872Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0345487Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0345775Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0346181Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp7yt4n2v
2022-12-01T10:49:12.0346607Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp7yt4n2v/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0347212Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T10:49:12.0347224Z 
2022-12-01T10:49:12.0347358Z Running tests...
2022-12-01T10:49:12.0347786Z ----------------------------------------------------------------------
2022-12-01T10:49:12.0348377Z   test_custom_stream_nested_multi (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8474
2022-12-01T10:49:12.0348716Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8475
2022-12-01T10:49:12.0349120Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8476
2022-12-01T10:49:12.0349507Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8477
2022-12-01T10:49:12.0350121Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0350383Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0351008Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0351274Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0351678Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzyenojck
2022-12-01T10:49:12.0352111Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzyenojck/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0352717Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0352979Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0353594Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0353883Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0354287Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfptfv4vj
2022-12-01T10:49:12.0354697Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfptfv4vj/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0355290Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0355547Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0356162Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0356457Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0356859Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0j1iru0z
2022-12-01T10:49:12.0357276Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0j1iru0z/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0357869Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0358145Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0358744Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0359036Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0359435Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6jx1kbqo
2022-12-01T10:49:12.0359852Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6jx1kbqo/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0360203Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:49:12.0360566Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:49:12.0360921Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:49:12.0361266Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:49:12.0361463Z skip: Need at least 2 CUDA devices (1.856s)
2022-12-01T10:49:12.0361489Z 
2022-12-01T10:49:12.0361886Z ----------------------------------------------------------------------
2022-12-01T10:49:12.0362039Z Ran 1 test in 1.857s
2022-12-01T10:49:12.0362049Z 
2022-12-01T10:49:12.0362203Z OK (skipped=1)
2022-12-01T10:49:12.0362212Z 
2022-12-01T10:49:12.0362395Z Generating XML reports...
2022-12-01T10:49:12.0363380Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201104511.xml
2022-12-01T10:49:12.0364036Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0364297Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0364913Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0365183Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0365579Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptrgopad1
2022-12-01T10:49:12.0366001Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptrgopad1/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0366586Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T10:49:12.0366598Z 
2022-12-01T10:49:12.0366756Z Running tests...
2022-12-01T10:49:12.0367176Z ----------------------------------------------------------------------
2022-12-01T10:49:12.0367743Z   test_device_map_cpu (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8529
2022-12-01T10:49:12.0368075Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8530
2022-12-01T10:49:12.0368407Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8531
2022-12-01T10:49:12.0368715Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8532
2022-12-01T10:49:12.0369307Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0369569Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0370187Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0370475Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0370880Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzqv5_abb
2022-12-01T10:49:12.0371301Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzqv5_abb/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0371907Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0372169Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0372827Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0373112Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0373510Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdqv95wnl
2022-12-01T10:49:12.0373964Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdqv95wnl/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0374566Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0374826Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0375447Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0375732Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0376111Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzk94nb7a
2022-12-01T10:49:12.0376526Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzk94nb7a/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0377119Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0377381Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0378066Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0378411Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0378813Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_ng86nq3
2022-12-01T10:49:12.0379233Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_ng86nq3/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0379592Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:49:12.0379927Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:49:12.0380286Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:49:12.0451410Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:49:12.0451574Z ok (2.153s)
2022-12-01T10:49:12.0451593Z 
2022-12-01T10:49:12.0452043Z ----------------------------------------------------------------------
2022-12-01T10:49:12.0452203Z Ran 1 test in 2.154s
2022-12-01T10:49:12.0452215Z 
2022-12-01T10:49:12.0452402Z OK
2022-12-01T10:49:12.0452414Z 
2022-12-01T10:49:12.0452596Z Generating XML reports...
2022-12-01T10:49:12.0453511Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201104515.xml
2022-12-01T10:49:12.0454123Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0454391Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0455020Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0455314Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0455720Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp67_s96z9
2022-12-01T10:49:12.0456140Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp67_s96z9/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0456730Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T10:49:12.0456742Z 
2022-12-01T10:49:12.0456895Z Running tests...
2022-12-01T10:49:12.0457298Z ----------------------------------------------------------------------
2022-12-01T10:49:12.0457903Z   test_device_map_cpu_to_gpu_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8752
2022-12-01T10:49:12.0458241Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8753
2022-12-01T10:49:12.0458573Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8754
2022-12-01T10:49:12.0458910Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8755
2022-12-01T10:49:12.0459511Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0459780Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0460547Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0460817Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0461223Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpaaioj3x9
2022-12-01T10:49:12.0461641Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpaaioj3x9/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0462250Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0462519Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0463296Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0463649Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0464051Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1yn5zxqs
2022-12-01T10:49:12.0464492Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1yn5zxqs/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0465080Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0465349Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0465974Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0466260Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0466667Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuejzq58z
2022-12-01T10:49:12.0467103Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuejzq58z/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0467710Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0467973Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0468594Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0468862Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0469263Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj8xukezf
2022-12-01T10:49:12.0469685Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj8xukezf/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0470046Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:49:12.0470396Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:49:12.0470750Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:49:12.0471096Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:49:12.0471309Z skip: Need at least 1 CUDA device (1.857s)
2022-12-01T10:49:12.0471319Z 
2022-12-01T10:49:12.0471720Z ----------------------------------------------------------------------
2022-12-01T10:49:12.0471879Z Ran 1 test in 1.857s
2022-12-01T10:49:12.0471889Z 
2022-12-01T10:49:12.0472044Z OK (skipped=1)
2022-12-01T10:49:12.0472053Z 
2022-12-01T10:49:12.0472230Z Generating XML reports...
2022-12-01T10:49:12.0473144Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201104518.xml
2022-12-01T10:49:12.0473748Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0474009Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0474619Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0474902Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0475281Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy5y3eyt6
2022-12-01T10:49:12.0475702Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy5y3eyt6/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0476294Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T10:49:12.0476304Z 
2022-12-01T10:49:12.0476460Z Running tests...
2022-12-01T10:49:12.0476884Z ----------------------------------------------------------------------
2022-12-01T10:49:12.0477537Z   test_device_map_cpu_to_gpu_non_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8807
2022-12-01T10:49:12.0477919Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8808
2022-12-01T10:49:12.0478252Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8809
2022-12-01T10:49:12.0478576Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8810
2022-12-01T10:49:12.0479155Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0479417Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0480037Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0480328Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0480724Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpblme3fu6
2022-12-01T10:49:12.0481150Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpblme3fu6/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0481741Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0481997Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0482594Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0482875Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0483274Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmk3cqcr1
2022-12-01T10:49:12.0483700Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmk3cqcr1/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0484307Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0484571Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0485189Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0485471Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0485870Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1xcdjwk5
2022-12-01T10:49:12.0486265Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1xcdjwk5/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0486858Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0487118Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0487740Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0488037Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0488439Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy7ijx4wr
2022-12-01T10:49:12.0488856Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy7ijx4wr/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0489209Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:49:12.0489542Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:49:12.0489895Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:49:12.0490245Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:49:12.0490458Z skip: Need at least 2 CUDA devices (1.855s)
2022-12-01T10:49:12.0490471Z 
2022-12-01T10:49:12.0490938Z ----------------------------------------------------------------------
2022-12-01T10:49:12.0491144Z Ran 1 test in 1.855s
2022-12-01T10:49:12.0491154Z 
2022-12-01T10:49:12.0491306Z OK (skipped=1)
2022-12-01T10:49:12.0491315Z 
2022-12-01T10:49:12.0491492Z Generating XML reports...
2022-12-01T10:49:12.0492496Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201104522.xml
2022-12-01T10:49:12.0493098Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0493368Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0493989Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0494275Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0494677Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwm9l32k7
2022-12-01T10:49:12.0495097Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwm9l32k7/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0495687Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T10:49:12.0495696Z 
2022-12-01T10:49:12.0495846Z Running tests...
2022-12-01T10:49:12.0496272Z ----------------------------------------------------------------------
2022-12-01T10:49:12.0496831Z   test_device_map_gpu_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8862
2022-12-01T10:49:12.0497162Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8863
2022-12-01T10:49:12.0497495Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8864
2022-12-01T10:49:12.0497821Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8865
2022-12-01T10:49:12.0498410Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0498675Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0499290Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0499580Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0499960Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkpisyu35
2022-12-01T10:49:12.0567176Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkpisyu35/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0567858Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0568121Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0568795Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0569086Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0569490Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpox3844l_
2022-12-01T10:49:12.0569906Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpox3844l_/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0570514Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0570757Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0571375Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0571668Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0572072Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1q7vsyvk
2022-12-01T10:49:12.0572682Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1q7vsyvk/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0573348Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0573615Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0574245Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0574509Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0574920Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdb049_ed
2022-12-01T10:49:12.0575340Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdb049_ed/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0575694Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:49:12.0576051Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:49:12.0576407Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:49:12.0576761Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:49:12.0576976Z skip: Need at least 2 CUDA devices (1.853s)
2022-12-01T10:49:12.0576988Z 
2022-12-01T10:49:12.0577415Z ----------------------------------------------------------------------
2022-12-01T10:49:12.0577553Z Ran 1 test in 1.854s
2022-12-01T10:49:12.0577565Z 
2022-12-01T10:49:12.0577716Z OK (skipped=1)
2022-12-01T10:49:12.0577725Z 
2022-12-01T10:49:12.0577898Z Generating XML reports...
2022-12-01T10:49:12.0578835Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201104525.xml
2022-12-01T10:49:12.0579446Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0579712Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0580335Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0580882Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0581293Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0o6u__16
2022-12-01T10:49:12.0581695Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0o6u__16/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0582292Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T10:49:12.0582305Z 
2022-12-01T10:49:12.0582454Z Running tests...
2022-12-01T10:49:12.0582874Z ----------------------------------------------------------------------
2022-12-01T10:49:12.0584602Z   test_device_map_gpu_default_to_non_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/80008 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.237s)
2022-12-01T10:49:12.0584619Z 
2022-12-01T10:49:12.0585036Z ----------------------------------------------------------------------
2022-12-01T10:49:12.0585193Z Ran 1 test in 0.237s
2022-12-01T10:49:12.0585204Z 
2022-12-01T10:49:12.0585351Z OK (skipped=1)
2022-12-01T10:49:12.0585361Z 
2022-12-01T10:49:12.0585543Z Generating XML reports...
2022-12-01T10:49:12.0586472Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201104528.xml
2022-12-01T10:49:12.0587059Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0587397Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0588079Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0588369Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0588765Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzttn4xu7
2022-12-01T10:49:12.0589187Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzttn4xu7/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0589774Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T10:49:12.0589783Z 
2022-12-01T10:49:12.0589937Z Running tests...
2022-12-01T10:49:12.0590341Z ----------------------------------------------------------------------
2022-12-01T10:49:12.0590918Z   test_device_map_gpu_mixed_1 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8927
2022-12-01T10:49:12.0591262Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8928
2022-12-01T10:49:12.0591584Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8929
2022-12-01T10:49:12.0591913Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8930
2022-12-01T10:49:12.0592514Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0592775Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0593402Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0593687Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0594073Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpel2t2qjf
2022-12-01T10:49:12.0594495Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpel2t2qjf/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0595093Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0595356Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0595974Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0596261Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0596661Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptr_knz0w
2022-12-01T10:49:12.0597077Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptr_knz0w/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0597688Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0597932Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0598545Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0598822Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0599217Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpakyk2qh5
2022-12-01T10:49:12.0599638Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpakyk2qh5/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0600235Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0600494Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0601113Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0601434Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0601839Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu8j2v47z
2022-12-01T10:49:12.0602310Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu8j2v47z/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0602675Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:49:12.0603031Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:49:12.0603390Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:49:12.0603742Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:49:12.0603966Z skip: Need at least 2 CUDA devices (1.951s)
2022-12-01T10:49:12.0603976Z 
2022-12-01T10:49:12.0604397Z ----------------------------------------------------------------------
2022-12-01T10:49:12.0604534Z Ran 1 test in 1.951s
2022-12-01T10:49:12.0604548Z 
2022-12-01T10:49:12.0604700Z OK (skipped=1)
2022-12-01T10:49:12.0604714Z 
2022-12-01T10:49:12.0604898Z Generating XML reports...
2022-12-01T10:49:12.0605819Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201104530.xml
2022-12-01T10:49:12.0606431Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0606688Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0607303Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0607588Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0607970Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqdyy7tf3
2022-12-01T10:49:12.0608395Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqdyy7tf3/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0608987Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T10:49:12.0608996Z 
2022-12-01T10:49:12.0609148Z Running tests...
2022-12-01T10:49:12.0609574Z ----------------------------------------------------------------------
2022-12-01T10:49:12.0610146Z   test_device_map_gpu_mixed_2 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8982
2022-12-01T10:49:12.0610470Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8983
2022-12-01T10:49:12.0610791Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8984
2022-12-01T10:49:12.0611119Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8985
2022-12-01T10:49:12.0611697Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0611959Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0612653Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0612941Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0613341Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0b0n1y75
2022-12-01T10:49:12.0613759Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0b0n1y75/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0614367Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0614629Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0615251Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0615563Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0616014Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwd28wtky
2022-12-01T10:49:12.0616450Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwd28wtky/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0617056Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0617318Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0617940Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0618237Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0618639Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph885xpxj
2022-12-01T10:49:12.0619050Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph885xpxj/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0619658Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0619922Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0676607Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0676920Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0677246Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp0zh209a
2022-12-01T10:49:12.0677442Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp0zh209a/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0677607Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:49:12.0677771Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:49:12.0677919Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:49:12.0678083Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:49:12.0678188Z skip: Need at least 2 CUDA devices (1.956s)
2022-12-01T10:49:12.0678198Z 
2022-12-01T10:49:12.0678421Z ----------------------------------------------------------------------
2022-12-01T10:49:12.0678497Z Ran 1 test in 1.956s
2022-12-01T10:49:12.0678502Z 
2022-12-01T10:49:12.0678573Z OK (skipped=1)
2022-12-01T10:49:12.0678578Z 
2022-12-01T10:49:12.0678660Z Generating XML reports...
2022-12-01T10:49:12.0679082Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201104534.xml
2022-12-01T10:49:12.0679358Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0679471Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0679751Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0679889Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0680072Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1rmwg3db
2022-12-01T10:49:12.0680267Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1rmwg3db/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0680535Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T10:49:12.0680540Z 
2022-12-01T10:49:12.0680612Z Running tests...
2022-12-01T10:49:12.0680807Z ----------------------------------------------------------------------
2022-12-01T10:49:12.0681060Z   test_device_map_gpu_mixed_3 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9037
2022-12-01T10:49:12.0681328Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9038
2022-12-01T10:49:12.0681518Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9039
2022-12-01T10:49:12.0681672Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9040
2022-12-01T10:49:12.0681946Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0682069Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0682352Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0682487Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0682670Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpoorc8rh4
2022-12-01T10:49:12.0682931Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0683055Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0683332Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0683470Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0683655Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkss4adav
2022-12-01T10:49:12.0683850Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpoorc8rh4/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0684047Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkss4adav/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0684320Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0684429Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0684710Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0684846Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0685028Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9bl_fm7p
2022-12-01T10:49:12.0685220Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9bl_fm7p/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0685489Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0685611Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0685890Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0686022Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0686195Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn4b6bjam
2022-12-01T10:49:12.0686394Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn4b6bjam/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0686557Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:49:12.0686722Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:49:12.0686883Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:49:12.0687043Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:49:12.0687145Z skip: Need at least 2 CUDA devices (1.952s)
2022-12-01T10:49:12.0687151Z 
2022-12-01T10:49:12.0687346Z ----------------------------------------------------------------------
2022-12-01T10:49:12.0687408Z Ran 1 test in 1.952s
2022-12-01T10:49:12.0687426Z 
2022-12-01T10:49:12.0687485Z OK (skipped=1)
2022-12-01T10:49:12.0687522Z 
2022-12-01T10:49:12.0687607Z Generating XML reports...
2022-12-01T10:49:12.0688059Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201104537.xml
2022-12-01T10:49:12.0688332Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0688456Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0688732Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0688866Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0689053Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyyghb9_9
2022-12-01T10:49:12.0689235Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyyghb9_9/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0689505Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T10:49:12.0689512Z 
2022-12-01T10:49:12.0689584Z Running tests...
2022-12-01T10:49:12.0689782Z ----------------------------------------------------------------------
2022-12-01T10:49:12.0690045Z   test_device_map_gpu_mixed_4 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9092
2022-12-01T10:49:12.0690201Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9093
2022-12-01T10:49:12.0690353Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9094
2022-12-01T10:49:12.0690505Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9095
2022-12-01T10:49:12.0690777Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0690887Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0691170Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0691305Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0691491Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzn2kdbab
2022-12-01T10:49:12.0691687Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzn2kdbab/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0691957Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0692078Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0692420Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0692558Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0692733Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbqefmqaj
2022-12-01T10:49:12.0692930Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbqefmqaj/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0693203Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0693324Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0693602Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0693734Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0693916Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfgjrqllk
2022-12-01T10:49:12.0694111Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfgjrqllk/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0694405Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0694557Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0694835Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0694972Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0695155Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7jfnqhyo
2022-12-01T10:49:12.0695349Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7jfnqhyo/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0695512Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:49:12.0695676Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:49:12.0695838Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:49:12.0695990Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:49:12.0696095Z skip: Need at least 2 CUDA devices (2.055s)
2022-12-01T10:49:12.0696100Z 
2022-12-01T10:49:12.0696296Z ----------------------------------------------------------------------
2022-12-01T10:49:12.0696370Z Ran 1 test in 2.055s
2022-12-01T10:49:12.0696375Z 
2022-12-01T10:49:12.0696446Z OK (skipped=1)
2022-12-01T10:49:12.0696450Z 
2022-12-01T10:49:12.0696533Z Generating XML reports...
2022-12-01T10:49:12.0696950Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201104541.xml
2022-12-01T10:49:12.0697224Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0697335Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0697615Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0697750Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0697933Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfaecyjue
2022-12-01T10:49:12.0698128Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfaecyjue/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0698395Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T10:49:12.0698401Z 
2022-12-01T10:49:12.0698472Z Running tests...
2022-12-01T10:49:12.0698669Z ----------------------------------------------------------------------
2022-12-01T10:49:12.0698934Z   test_device_map_gpu_mixed_5 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9147
2022-12-01T10:49:12.0699077Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9148
2022-12-01T10:49:12.0699228Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9149
2022-12-01T10:49:12.0699380Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9150
2022-12-01T10:49:12.0699652Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0699775Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0700056Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0700192Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0700498Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5zldmd41
2022-12-01T10:49:12.0700696Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5zldmd41/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0701010Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0701168Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0701446Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0701580Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0701761Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpudjozu8q
2022-12-01T10:49:12.0701960Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpudjozu8q/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0702230Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0702350Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0702616Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0702749Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0702933Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd60w9ao_
2022-12-01T10:49:12.0703127Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd60w9ao_/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0703396Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0703519Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0703797Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0703932Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0704114Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1ethxvzm
2022-12-01T10:49:12.0704295Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1ethxvzm/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0704459Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:49:12.0704621Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:49:12.0704782Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:49:12.0704943Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:49:12.0705044Z skip: Need at least 2 CUDA devices (1.862s)
2022-12-01T10:49:12.0705050Z 
2022-12-01T10:49:12.0705248Z ----------------------------------------------------------------------
2022-12-01T10:49:12.0705323Z Ran 1 test in 1.862s
2022-12-01T10:49:12.0705328Z 
2022-12-01T10:49:12.0705385Z OK (skipped=1)
2022-12-01T10:49:12.0705401Z 
2022-12-01T10:49:12.0705471Z Generating XML reports...
2022-12-01T10:49:12.0705889Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201104544.xml
2022-12-01T10:49:12.0706163Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0706287Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0706566Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0706701Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0706887Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxg18pw7h
2022-12-01T10:49:12.0707083Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxg18pw7h/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0707370Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T10:49:12.0707389Z 
2022-12-01T10:49:12.0707475Z Running tests...
2022-12-01T10:49:12.0707671Z ----------------------------------------------------------------------
2022-12-01T10:49:12.0707936Z   test_device_map_gpu_mixed_6 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9202
2022-12-01T10:49:12.0708091Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9203
2022-12-01T10:49:12.0708243Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9204
2022-12-01T10:49:12.0708394Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9205
2022-12-01T10:49:12.0708666Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0708789Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0709058Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0709194Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0709379Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8dbnzpey
2022-12-01T10:49:12.0709574Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8dbnzpey/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0709847Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0709968Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0710245Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0710379Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0710550Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp28f8_8gb
2022-12-01T10:49:12.0710741Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp28f8_8gb/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0711011Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0711132Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0711410Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0711544Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0711728Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe5jnlgbv
2022-12-01T10:49:12.0711921Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe5jnlgbv/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0712188Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0712299Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0712579Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0712711Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0712893Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprtzlw8gv
2022-12-01T10:49:12.0713085Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprtzlw8gv/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0713248Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:49:12.0713410Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:49:12.0713571Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:49:12.0713756Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:49:12.0713886Z skip: Need at least 2 CUDA devices (1.853s)
2022-12-01T10:49:12.0713892Z 
2022-12-01T10:49:12.0714087Z ----------------------------------------------------------------------
2022-12-01T10:49:12.0714161Z Ran 1 test in 1.853s
2022-12-01T10:49:12.0714166Z 
2022-12-01T10:49:12.0714236Z OK (skipped=1)
2022-12-01T10:49:12.0714242Z 
2022-12-01T10:49:12.0714325Z Generating XML reports...
2022-12-01T10:49:12.0714749Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201104548.xml
2022-12-01T10:49:12.0715022Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0715146Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0715415Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0715549Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0715734Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpveiqhguk
2022-12-01T10:49:12.0715930Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpveiqhguk/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0716196Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T10:49:12.0716201Z 
2022-12-01T10:49:12.0716273Z Running tests...
2022-12-01T10:49:12.0716469Z ----------------------------------------------------------------------
2022-12-01T10:49:12.0716730Z   test_device_map_gpu_mixed_7 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9257
2022-12-01T10:49:12.0716884Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9258
2022-12-01T10:49:12.0717025Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9259
2022-12-01T10:49:12.0717176Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9260
2022-12-01T10:49:12.0717446Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0717568Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0717844Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0717976Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0718160Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjxzrrpi_
2022-12-01T10:49:12.0718358Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjxzrrpi_/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0718616Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0718738Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0719020Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0719152Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0719333Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkd41x7gu
2022-12-01T10:49:12.0719524Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkd41x7gu/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0719794Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0719914Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0720191Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0720344Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0720557Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppmhxlk1b
2022-12-01T10:49:12.0720751Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppmhxlk1b/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0721018Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0721141Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0721419Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0721551Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0721733Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzu3ip068
2022-12-01T10:49:12.0721930Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzu3ip068/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0722083Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:49:12.0722244Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:49:12.0722407Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:49:12.0722564Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:49:12.0722666Z skip: Need at least 2 CUDA devices (1.857s)
2022-12-01T10:49:12.0722671Z 
2022-12-01T10:49:12.0722866Z ----------------------------------------------------------------------
2022-12-01T10:49:12.0722940Z Ran 1 test in 1.857s
2022-12-01T10:49:12.0722948Z 
2022-12-01T10:49:12.0723019Z OK (skipped=1)
2022-12-01T10:49:12.0723024Z 
2022-12-01T10:49:12.0723093Z Generating XML reports...
2022-12-01T10:49:12.0723513Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201104551.xml
2022-12-01T10:49:12.0723787Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0723913Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0724191Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0724323Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0724506Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo7afinvt
2022-12-01T10:49:12.0724701Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo7afinvt/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0724967Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T10:49:12.0724972Z 
2022-12-01T10:49:12.0725033Z Running tests...
2022-12-01T10:49:12.0725226Z ----------------------------------------------------------------------
2022-12-01T10:49:12.0725493Z   test_device_map_gpu_mixed_8 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9312
2022-12-01T10:49:12.0725647Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9313
2022-12-01T10:49:12.0725798Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9314
2022-12-01T10:49:12.0725948Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9315
2022-12-01T10:49:12.0726221Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0726343Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0726611Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0726790Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0727002Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsnk02bbv
2022-12-01T10:49:12.0727195Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsnk02bbv/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0727467Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0727590Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0727865Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0727999Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0728183Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_d6titah
2022-12-01T10:49:12.0728367Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_d6titah/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0728640Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0728762Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0729040Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0729178Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0729361Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbvx_zb9t
2022-12-01T10:49:12.0729552Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbvx_zb9t/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0729821Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0729946Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0730211Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0730348Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0730528Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_ja96ndk
2022-12-01T10:49:12.0730720Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_ja96ndk/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0730884Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:49:12.0731048Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:49:12.0731207Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:49:12.0731368Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:49:12.0731460Z skip: Need at least 2 CUDA devices (1.854s)
2022-12-01T10:49:12.0731479Z 
2022-12-01T10:49:12.0731661Z ----------------------------------------------------------------------
2022-12-01T10:49:12.0731737Z Ran 1 test in 1.854s
2022-12-01T10:49:12.0731742Z 
2022-12-01T10:49:12.0731812Z OK (skipped=1)
2022-12-01T10:49:12.0731817Z 
2022-12-01T10:49:12.0731900Z Generating XML reports...
2022-12-01T10:49:12.0732382Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201104554.xml
2022-12-01T10:49:12.0732661Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0732784Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0733060Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0733265Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0733452Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsdncffwr
2022-12-01T10:49:12.0733677Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsdncffwr/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0733948Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T10:49:12.0733952Z 
2022-12-01T10:49:12.0734025Z Running tests...
2022-12-01T10:49:12.0734220Z ----------------------------------------------------------------------
2022-12-01T10:49:12.0734489Z   test_device_map_gpu_mixed_self_1 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9367
2022-12-01T10:49:12.0734643Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9368
2022-12-01T10:49:12.0734794Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9369
2022-12-01T10:49:12.0734934Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9370
2022-12-01T10:49:12.0735207Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0735332Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0735611Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0735745Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0735930Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpynntom8y
2022-12-01T10:49:12.0736126Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpynntom8y/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0736400Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0736512Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0736789Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0736925Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0737107Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpynf4kvzv
2022-12-01T10:49:12.0737301Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpynf4kvzv/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0737569Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0737691Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0737967Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0738101Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0738273Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9fe49ayf
2022-12-01T10:49:12.0738544Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0738665Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0738943Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0739076Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0739258Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp34oecfj8
2022-12-01T10:49:12.0739451Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9fe49ayf/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0739644Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp34oecfj8/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0739828Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:49:12.0740019Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:49:12.0740175Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:49:12.0740336Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:49:12.0740558Z skip: Need at least 2 CUDA devices (1.855s)
2022-12-01T10:49:12.0740564Z 
2022-12-01T10:49:12.0740764Z ----------------------------------------------------------------------
2022-12-01T10:49:12.0740837Z Ran 1 test in 1.855s
2022-12-01T10:49:12.0740842Z 
2022-12-01T10:49:12.0740913Z OK (skipped=1)
2022-12-01T10:49:12.0740919Z 
2022-12-01T10:49:12.0741001Z Generating XML reports...
2022-12-01T10:49:12.0741411Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201104558.xml
2022-12-01T10:49:12.0741683Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0741808Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0742088Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0742223Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0742406Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp32z0yake
2022-12-01T10:49:12.0742599Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp32z0yake/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0742869Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T10:49:12.0742874Z 
2022-12-01T10:49:12.0742964Z Running tests...
2022-12-01T10:49:12.0743149Z ----------------------------------------------------------------------
2022-12-01T10:49:12.0743419Z   test_device_map_gpu_mixed_self_2 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9422
2022-12-01T10:49:12.0743573Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9423
2022-12-01T10:49:12.0743723Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9424
2022-12-01T10:49:12.0743874Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9425
2022-12-01T10:49:12.0744145Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0744267Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0744547Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0744673Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0744857Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz6z7h0a1
2022-12-01T10:49:12.0745054Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz6z7h0a1/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0745325Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0745447Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0745728Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0745863Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0746046Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1vvqqbu2
2022-12-01T10:49:12.0746241Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1vvqqbu2/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0746547Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0746703Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0746981Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0747116Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0747299Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkpnbk2bs
2022-12-01T10:49:12.0747494Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkpnbk2bs/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0747761Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0747883Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0748150Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0748285Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0748465Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc_mej304
2022-12-01T10:49:12.0748655Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc_mej304/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0748817Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:49:12.0748979Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:49:12.0749139Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:49:12.0749299Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:49:12.0749401Z skip: Need at least 2 CUDA devices (1.854s)
2022-12-01T10:49:12.0749406Z 
2022-12-01T10:49:12.0749590Z ----------------------------------------------------------------------
2022-12-01T10:49:12.0749668Z Ran 1 test in 1.855s
2022-12-01T10:49:12.0749673Z 
2022-12-01T10:49:12.0749743Z OK (skipped=1)
2022-12-01T10:49:12.0749748Z 
2022-12-01T10:49:12.0749830Z Generating XML reports...
2022-12-01T10:49:12.0750245Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201104601.xml
2022-12-01T10:49:12.0750515Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0750637Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0750914Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0751048Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0751223Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpeuo3l74f
2022-12-01T10:49:12.0751419Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpeuo3l74f/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0751684Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T10:49:12.0751689Z 
2022-12-01T10:49:12.0751764Z Running tests...
2022-12-01T10:49:12.0751956Z ----------------------------------------------------------------------
2022-12-01T10:49:12.0752222Z   test_device_map_gpu_mixed_self_3 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9477
2022-12-01T10:49:12.0752377Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9478
2022-12-01T10:49:12.0752528Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9479
2022-12-01T10:49:12.0752698Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9480
2022-12-01T10:49:12.0752972Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0753125Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0753407Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0753541Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0753726Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv8e3yctc
2022-12-01T10:49:12.0753921Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv8e3yctc/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0754191Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0754313Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0754579Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0754714Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0754898Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpklhqjkip
2022-12-01T10:49:12.0755094Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpklhqjkip/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0755363Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0755485Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0755762Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0755897Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0756082Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvwqwlf0i
2022-12-01T10:49:12.0756264Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvwqwlf0i/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0756532Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0756655Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0756934Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0757066Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0757248Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7wfyxmag
2022-12-01T10:49:12.0757441Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7wfyxmag/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0757608Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:49:12.0757758Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:49:12.0757921Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:49:12.0758080Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:49:12.0758183Z skip: Need at least 2 CUDA devices (1.953s)
2022-12-01T10:49:12.0758189Z 
2022-12-01T10:49:12.0758382Z ----------------------------------------------------------------------
2022-12-01T10:49:12.0758457Z Ran 1 test in 1.953s
2022-12-01T10:49:12.0758461Z 
2022-12-01T10:49:12.0758531Z OK (skipped=1)
2022-12-01T10:49:12.0758535Z 
2022-12-01T10:49:12.0758619Z Generating XML reports...
2022-12-01T10:49:12.0759038Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201104605.xml
2022-12-01T10:49:12.0759334Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0759485Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0759765Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0759898Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0760082Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplc6mwebc
2022-12-01T10:49:12.0760277Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplc6mwebc/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0760544Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T10:49:12.0760548Z 
2022-12-01T10:49:12.0760620Z Running tests...
2022-12-01T10:49:12.0760801Z ----------------------------------------------------------------------
2022-12-01T10:49:12.0761069Z   test_device_map_gpu_mixed_self_4 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9532
2022-12-01T10:49:12.0761225Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9533
2022-12-01T10:49:12.0761380Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9534
2022-12-01T10:49:12.0761528Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9535
2022-12-01T10:49:12.0761801Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0761924Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0762204Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0762339Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0762511Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4scie85e
2022-12-01T10:49:12.0762706Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4scie85e/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0762975Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0763097Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0763374Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0763508Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0763690Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpszp0tnxj
2022-12-01T10:49:12.0763885Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpszp0tnxj/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0764158Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0764270Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0764547Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0764683Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0764866Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9ke3sfqn
2022-12-01T10:49:12.0765058Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9ke3sfqn/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0765328Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0765450Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0765755Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0765881Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0766096Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjqnpi7wr
2022-12-01T10:49:12.0766288Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjqnpi7wr/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0766452Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:49:12.0766614Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:49:12.0766774Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:49:12.0766935Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:49:12.0767039Z skip: Need at least 2 CUDA devices (1.854s)
2022-12-01T10:49:12.0767044Z 
2022-12-01T10:49:12.0767240Z ----------------------------------------------------------------------
2022-12-01T10:49:12.0767302Z Ran 1 test in 1.854s
2022-12-01T10:49:12.0767309Z 
2022-12-01T10:49:12.0767379Z OK (skipped=1)
2022-12-01T10:49:12.0767383Z 
2022-12-01T10:49:12.0767466Z Generating XML reports...
2022-12-01T10:49:12.0767879Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201104608.xml
2022-12-01T10:49:12.0768148Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0768271Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0768548Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0768682Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0768855Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxrfen3hw
2022-12-01T10:49:12.0769050Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxrfen3hw/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0769318Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T10:49:12.0769324Z 
2022-12-01T10:49:12.0769396Z Running tests...
2022-12-01T10:49:12.0769588Z ----------------------------------------------------------------------
2022-12-01T10:49:12.0769855Z   test_device_map_gpu_mixed_self_5 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9587
2022-12-01T10:49:12.0770016Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9588
2022-12-01T10:49:12.0770168Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9589
2022-12-01T10:49:12.0770317Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9590
2022-12-01T10:49:12.0770583Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0770707Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0770986Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0771120Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0771303Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplu57t34t
2022-12-01T10:49:12.0771499Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplu57t34t/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0771770Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0771892Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0772213Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0772438Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0772626Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq1vvr_2l
2022-12-01T10:49:12.0772818Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq1vvr_2l/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0773094Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0773215Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0773492Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0773625Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0773807Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6ceamuz4
2022-12-01T10:49:12.0773989Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6ceamuz4/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0774257Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0774379Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0774658Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0774792Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0774975Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8k3sxb0p
2022-12-01T10:49:12.0775167Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8k3sxb0p/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0775331Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:49:12.0775496Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:49:12.0775647Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:49:12.0775808Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:49:12.0775911Z skip: Need at least 2 CUDA devices (1.956s)
2022-12-01T10:49:12.0775916Z 
2022-12-01T10:49:12.0776112Z ----------------------------------------------------------------------
2022-12-01T10:49:12.0776186Z Ran 1 test in 1.957s
2022-12-01T10:49:12.0776191Z 
2022-12-01T10:49:12.0776261Z OK (skipped=1)
2022-12-01T10:49:12.0776266Z 
2022-12-01T10:49:12.0776349Z Generating XML reports...
2022-12-01T10:49:12.0776766Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201104612.xml
2022-12-01T10:49:12.0777041Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0777153Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0777432Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0777567Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0777751Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe5tu1jhb
2022-12-01T10:49:12.0777944Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe5tu1jhb/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0778213Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T10:49:12.0778218Z 
2022-12-01T10:49:12.0778290Z Running tests...
2022-12-01T10:49:12.0778484Z ----------------------------------------------------------------------
2022-12-01T10:49:12.0778779Z   test_device_map_gpu_mixed_self_6 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9642
2022-12-01T10:49:12.0778959Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9643
2022-12-01T10:49:12.0779109Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9644
2022-12-01T10:49:12.0779259Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9645
2022-12-01T10:49:12.0779534Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0779656Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0779937Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0780072Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0780258Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyb_pqlp3
2022-12-01T10:49:12.0780552Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyb_pqlp3/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0780829Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0780951Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0781228Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0781361Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0781544Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpblgip_u2
2022-12-01T10:49:12.0781738Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpblgip_u2/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0782009Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0782120Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0782400Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0782534Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0782716Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnnx_kxwr
2022-12-01T10:49:12.0782909Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnnx_kxwr/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0783178Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0783301Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0783583Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0783721Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0783894Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9m109rrf
2022-12-01T10:49:12.0784087Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9m109rrf/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0784252Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:49:12.0784416Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:49:12.0784575Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:49:12.0784736Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:49:12.0784838Z skip: Need at least 2 CUDA devices (1.853s)
2022-12-01T10:49:12.0784846Z 
2022-12-01T10:49:12.0785042Z ----------------------------------------------------------------------
2022-12-01T10:49:12.0785154Z Ran 1 test in 1.853s
2022-12-01T10:49:12.0785172Z 
2022-12-01T10:49:12.0785264Z OK (skipped=1)
2022-12-01T10:49:12.0785269Z 
2022-12-01T10:49:12.0785352Z Generating XML reports...
2022-12-01T10:49:12.0785772Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201104615.xml
2022-12-01T10:49:12.0786044Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0786168Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0786447Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0786581Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0786764Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2e900jm5
2022-12-01T10:49:12.0786947Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2e900jm5/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0787216Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T10:49:12.0787221Z 
2022-12-01T10:49:12.0787294Z Running tests...
2022-12-01T10:49:12.0787491Z ----------------------------------------------------------------------
2022-12-01T10:49:12.0787760Z   test_device_map_gpu_mixed_self_7 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9697
2022-12-01T10:49:12.0787914Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9698
2022-12-01T10:49:12.0788069Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9699
2022-12-01T10:49:12.0788220Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9700
2022-12-01T10:49:12.0788496Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0788607Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0788886Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0789021Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0789205Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp54dxbeuo
2022-12-01T10:49:12.0789398Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp54dxbeuo/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0789669Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0789791Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0790068Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0790205Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0790379Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz1atkgpk
2022-12-01T10:49:12.0790648Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0790769Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0791051Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0791185Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0791368Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplmipqwx0
2022-12-01T10:49:12.0791563Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz1atkgpk/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0791785Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplmipqwx0/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0792067Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0792192Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0792470Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0792604Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0792785Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpll36iw6a
2022-12-01T10:49:12.0792975Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpll36iw6a/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0793138Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:49:12.0793300Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:49:12.0793463Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:49:12.0793613Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:49:12.0793714Z skip: Need at least 2 CUDA devices (1.958s)
2022-12-01T10:49:12.0793719Z 
2022-12-01T10:49:12.0793918Z ----------------------------------------------------------------------
2022-12-01T10:49:12.0793993Z Ran 1 test in 1.958s
2022-12-01T10:49:12.0793997Z 
2022-12-01T10:49:12.0794068Z OK (skipped=1)
2022-12-01T10:49:12.0794073Z 
2022-12-01T10:49:12.0794154Z Generating XML reports...
2022-12-01T10:49:12.0794571Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201104618.xml
2022-12-01T10:49:12.0794845Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0794957Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0795238Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0795374Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0795556Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxjrmkibr
2022-12-01T10:49:12.0795751Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxjrmkibr/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0796021Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T10:49:12.0796026Z 
2022-12-01T10:49:12.0796097Z Running tests...
2022-12-01T10:49:12.0796292Z ----------------------------------------------------------------------
2022-12-01T10:49:12.0796564Z   test_device_map_gpu_mixed_self_8 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9752
2022-12-01T10:49:12.0796706Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9753
2022-12-01T10:49:12.0796861Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9754
2022-12-01T10:49:12.0797011Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9755
2022-12-01T10:49:12.0797285Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0797412Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0797690Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0797824Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0798006Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu3_c_j1q
2022-12-01T10:49:12.0798234Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu3_c_j1q/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0798516Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0798637Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0798914Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0799049Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0799231Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkvpo979y
2022-12-01T10:49:12.0799426Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkvpo979y/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0799693Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0799818Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0800081Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0800216Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0800400Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwltob5ib
2022-12-01T10:49:12.0800595Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwltob5ib/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0800863Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0800984Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0801262Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0801396Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0801579Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplxt14xfn
2022-12-01T10:49:12.0801763Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplxt14xfn/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0801928Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:49:12.0802091Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:49:12.0802255Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:49:12.0802413Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:49:12.0802517Z skip: Need at least 2 CUDA devices (1.855s)
2022-12-01T10:49:12.0802523Z 
2022-12-01T10:49:12.0802714Z ----------------------------------------------------------------------
2022-12-01T10:49:12.0802790Z Ran 1 test in 1.855s
2022-12-01T10:49:12.0802795Z 
2022-12-01T10:49:12.0802855Z OK (skipped=1)
2022-12-01T10:49:12.0802872Z 
2022-12-01T10:49:12.0802945Z Generating XML reports...
2022-12-01T10:49:12.0803360Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201104622.xml
2022-12-01T10:49:12.0803634Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0803757Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0804037Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0804172Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0804354Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprkw6uaj0
2022-12-01T10:49:12.0804575Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprkw6uaj0/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0804832Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T10:49:12.0804879Z 
2022-12-01T10:49:12.0804939Z Running tests...
2022-12-01T10:49:12.0805135Z ----------------------------------------------------------------------
2022-12-01T10:49:12.0805413Z   test_device_map_gpu_non_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9807
2022-12-01T10:49:12.0805568Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9808
2022-12-01T10:49:12.0805721Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9809
2022-12-01T10:49:12.0805859Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9810
2022-12-01T10:49:12.0806135Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0806261Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0806543Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0806679Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0806866Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdeq632vn
2022-12-01T10:49:12.0807063Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdeq632vn/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0807335Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0807492Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0807754Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0807889Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0808075Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmd76r05t
2022-12-01T10:49:12.0808271Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmd76r05t/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0808544Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0808668Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0808947Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0809082Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0809267Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu8jaogkn
2022-12-01T10:49:12.0809450Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu8jaogkn/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0809720Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0809845Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0810125Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0810259Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0810443Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphf8cqh9x
2022-12-01T10:49:12.0810635Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphf8cqh9x/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0810799Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:49:12.0810950Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:49:12.0811150Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:49:12.0811372Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:49:12.0811530Z skip: Need at least 2 CUDA devices (1.955s)
2022-12-01T10:49:12.0811537Z 
2022-12-01T10:49:12.0811849Z ----------------------------------------------------------------------
2022-12-01T10:49:12.0811965Z Ran 1 test in 1.955s
2022-12-01T10:49:12.0811973Z 
2022-12-01T10:49:12.0812077Z OK (skipped=1)
2022-12-01T10:49:12.0812083Z 
2022-12-01T10:49:12.0812208Z Generating XML reports...
2022-12-01T10:49:12.0812754Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201104625.xml
2022-12-01T10:49:12.0813016Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0813140Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0813428Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0813566Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0813750Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwlw5ga8d
2022-12-01T10:49:12.0813950Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwlw5ga8d/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0814220Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T10:49:12.0814225Z 
2022-12-01T10:49:12.0814302Z Running tests...
2022-12-01T10:49:12.0814486Z ----------------------------------------------------------------------
2022-12-01T10:49:12.0814771Z   test_device_map_gpu_non_default_to_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9862
2022-12-01T10:49:12.0814930Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9863
2022-12-01T10:49:12.0815085Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9864
2022-12-01T10:49:12.0815239Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9865
2022-12-01T10:49:12.0815513Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0815636Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0815917Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0816051Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0816222Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2fwbu15w
2022-12-01T10:49:12.0816419Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2fwbu15w/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0816690Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0816819Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0817097Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0817232Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0817416Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj6jvo2p5
2022-12-01T10:49:12.0817610Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj6jvo2p5/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0817880Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0817989Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0818337Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0818506Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0818691Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4modp1tb
2022-12-01T10:49:12.0818887Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4modp1tb/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0819158Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0819282Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0819560Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0819681Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0819868Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg_3w3qqm
2022-12-01T10:49:12.0820064Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg_3w3qqm/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0820229Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:49:12.0820511Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:49:12.0820679Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:49:12.0820840Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:49:12.0820944Z skip: Need at least 2 CUDA devices (1.858s)
2022-12-01T10:49:12.0820950Z 
2022-12-01T10:49:12.0821149Z ----------------------------------------------------------------------
2022-12-01T10:49:12.0821211Z Ran 1 test in 1.858s
2022-12-01T10:49:12.0821216Z 
2022-12-01T10:49:12.0821288Z OK (skipped=1)
2022-12-01T10:49:12.0821293Z 
2022-12-01T10:49:12.0821380Z Generating XML reports...
2022-12-01T10:49:12.0821803Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201104629.xml
2022-12-01T10:49:12.0822079Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0822204Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0822488Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0822623Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0822811Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg0vo95uo
2022-12-01T10:49:12.0822993Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg0vo95uo/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0823263Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T10:49:12.0823269Z 
2022-12-01T10:49:12.0823342Z Running tests...
2022-12-01T10:49:12.0823534Z ----------------------------------------------------------------------
2022-12-01T10:49:12.0823809Z   test_device_map_gpu_to_cpu_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9917
2022-12-01T10:49:12.0823963Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9918
2022-12-01T10:49:12.0824115Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9919
2022-12-01T10:49:12.0824265Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9920
2022-12-01T10:49:12.0824524Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0824647Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0824984Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0825152Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0825336Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz_ago36y
2022-12-01T10:49:12.0825531Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz_ago36y/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0825803Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0825927Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0826205Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0826327Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0826514Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo1gc66ax
2022-12-01T10:49:12.0826712Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo1gc66ax/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0826984Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0827107Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0827386Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0827520Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0827704Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwj8s30fg
2022-12-01T10:49:12.0827947Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwj8s30fg/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0828219Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0828345Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0828628Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0828763Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0828947Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgc4bz8rg
2022-12-01T10:49:12.0829140Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgc4bz8rg/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0829304Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:49:12.0829469Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:49:12.0829617Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:49:12.0829780Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:49:12.0829885Z skip: Need at least 1 CUDA device (1.954s)
2022-12-01T10:49:12.0829890Z 
2022-12-01T10:49:12.0830084Z ----------------------------------------------------------------------
2022-12-01T10:49:12.0830159Z Ran 1 test in 1.954s
2022-12-01T10:49:12.0830164Z 
2022-12-01T10:49:12.0830235Z OK (skipped=1)
2022-12-01T10:49:12.0830239Z 
2022-12-01T10:49:12.0830323Z Generating XML reports...
2022-12-01T10:49:12.0830737Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201104632.xml
2022-12-01T10:49:12.0831010Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0831121Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0831429Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0831591Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0831775Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpja64694a
2022-12-01T10:49:12.0831969Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpja64694a/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0832237Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T10:49:12.0832244Z 
2022-12-01T10:49:12.0832316Z Running tests...
2022-12-01T10:49:12.0832514Z ----------------------------------------------------------------------
2022-12-01T10:49:12.0832777Z   test_device_map_gpu_to_cpu_non_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9972
2022-12-01T10:49:12.0832932Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9973
2022-12-01T10:49:12.0833082Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9974
2022-12-01T10:49:12.0833235Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9975
2022-12-01T10:49:12.0833506Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0833629Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0833909Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0834044Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0834226Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjn_0rkxo
2022-12-01T10:49:12.0834407Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjn_0rkxo/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0834679Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0834804Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0835081Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0835216Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0835399Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg31tj_1d
2022-12-01T10:49:12.0835592Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg31tj_1d/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0835859Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0835984Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0836250Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0836388Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0836573Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo34iu8d9
2022-12-01T10:49:12.0836764Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo34iu8d9/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0837029Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0837153Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0837431Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0837563Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0837733Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe2sjq4u2
2022-12-01T10:49:12.0837962Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe2sjq4u2/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0838155Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:49:12.0838319Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:49:12.0838481Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:49:12.0838640Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:49:12.0838743Z skip: Need at least 2 CUDA devices (1.965s)
2022-12-01T10:49:12.0838749Z 
2022-12-01T10:49:12.0838943Z ----------------------------------------------------------------------
2022-12-01T10:49:12.0839017Z Ran 1 test in 1.966s
2022-12-01T10:49:12.0839022Z 
2022-12-01T10:49:12.0839080Z OK (skipped=1)
2022-12-01T10:49:12.0839085Z 
2022-12-01T10:49:12.0839168Z Generating XML reports...
2022-12-01T10:49:12.0839590Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201104636.xml
2022-12-01T10:49:12.0839865Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0839990Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0840267Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0840403Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0840587Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjycbls0j
2022-12-01T10:49:12.0840770Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjycbls0j/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0841044Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T10:49:12.0841049Z 
2022-12-01T10:49:12.0841124Z Running tests...
2022-12-01T10:49:12.0841320Z ----------------------------------------------------------------------
2022-12-01T10:49:12.0841582Z   test_device_maps_gpu (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10027
2022-12-01T10:49:12.0841742Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10028
2022-12-01T10:49:12.0841894Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10029
2022-12-01T10:49:12.0842046Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10030
2022-12-01T10:49:12.0842319Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0842427Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0842708Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0842844Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0843029Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpouriyfnp
2022-12-01T10:49:12.0843226Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpouriyfnp/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0843497Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0843621Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0843902Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0844035Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0844236Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfk2_dpn_
2022-12-01T10:49:12.0844430Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfk2_dpn_/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0844729Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0844851Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0845125Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0845260Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0845442Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpne0ogc8y
2022-12-01T10:49:12.0845634Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpne0ogc8y/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0845887Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0846013Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0846297Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0846431Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0846613Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsv_t1h_y
2022-12-01T10:49:12.0846804Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsv_t1h_y/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0846968Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:49:12.0847134Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:49:12.0847294Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:49:12.0847443Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:49:12.0847549Z skip: Need at least 2 CUDA devices (1.955s)
2022-12-01T10:49:12.0847555Z 
2022-12-01T10:49:12.0847751Z ----------------------------------------------------------------------
2022-12-01T10:49:12.0847826Z Ran 1 test in 1.955s
2022-12-01T10:49:12.0847831Z 
2022-12-01T10:49:12.0847901Z OK (skipped=1)
2022-12-01T10:49:12.0847906Z 
2022-12-01T10:49:12.0847988Z Generating XML reports...
2022-12-01T10:49:12.0848402Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201104639.xml
2022-12-01T10:49:12.0848676Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0848786Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0849068Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0849202Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0849388Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk00fq03k
2022-12-01T10:49:12.0849584Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk00fq03k/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0849854Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T10:49:12.0849859Z 
2022-12-01T10:49:12.0849930Z Running tests...
2022-12-01T10:49:12.0850126Z ----------------------------------------------------------------------
2022-12-01T10:49:12.0850394Z   test_device_maps_in_options (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10082
2022-12-01T10:49:12.0850537Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10083
2022-12-01T10:49:12.0850720Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10084
2022-12-01T10:49:12.0850899Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10085
2022-12-01T10:49:12.0851169Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0851293Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0851572Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0851708Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0851892Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpiogl459x
2022-12-01T10:49:12.0852163Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0852273Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0852637Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0852778Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0852966Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4hvp1q_v
2022-12-01T10:49:12.0853166Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpiogl459x/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0853359Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4hvp1q_v/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0853632Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0853756Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0854017Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0854154Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0854347Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy8ld687p
2022-12-01T10:49:12.0854618Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0854742Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0855019Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0855157Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0855340Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu9_q1msw
2022-12-01T10:49:12.0855535Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy8ld687p/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0855715Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu9_q1msw/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0855880Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:49:12.0856041Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:49:12.0856203Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:49:12.0856366Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:49:12.0856468Z skip: Need at least 2 CUDA devices (1.957s)
2022-12-01T10:49:12.0856473Z 
2022-12-01T10:49:12.0856672Z ----------------------------------------------------------------------
2022-12-01T10:49:12.0856747Z Ran 1 test in 1.957s
2022-12-01T10:49:12.0856751Z 
2022-12-01T10:49:12.0856808Z OK (skipped=1)
2022-12-01T10:49:12.0856827Z 
2022-12-01T10:49:12.0856898Z Generating XML reports...
2022-12-01T10:49:12.0857349Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201104643.xml
2022-12-01T10:49:12.0857660Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0857787Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0858068Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0858204Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0858388Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6wuigtvc
2022-12-01T10:49:12.0858580Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6wuigtvc/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0858835Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T10:49:12.0858854Z 
2022-12-01T10:49:12.0858916Z Running tests...
2022-12-01T10:49:12.0859111Z ----------------------------------------------------------------------
2022-12-01T10:49:12.0859398Z   test_device_maps_invalid_max_local_device (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10137
2022-12-01T10:49:12.0859556Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10138
2022-12-01T10:49:12.0859712Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10139
2022-12-01T10:49:12.0859867Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10140
2022-12-01T10:49:12.0860144Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0860268Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0860675Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0860814Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0861000Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5x6syoc1
2022-12-01T10:49:12.0861195Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5x6syoc1/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0861470Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0861594Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0861872Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0862009Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0862178Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgktag9mt
2022-12-01T10:49:12.0862376Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgktag9mt/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0862651Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0862773Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0863053Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0863188Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0863376Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfkbk9vty
2022-12-01T10:49:12.0863572Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfkbk9vty/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0863846Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0864027Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0864311Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0864484Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0864667Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_pkx7xau
2022-12-01T10:49:12.0864862Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_pkx7xau/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0865028Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:49:12.0865191Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:49:12.0865355Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:49:12.0865518Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:49:12.0865608Z skip: Need at least 1 CUDA device (1.758s)
2022-12-01T10:49:12.0865617Z 
2022-12-01T10:49:12.0865817Z ----------------------------------------------------------------------
2022-12-01T10:49:12.0865893Z Ran 1 test in 1.758s
2022-12-01T10:49:12.0865898Z 
2022-12-01T10:49:12.0865969Z OK (skipped=1)
2022-12-01T10:49:12.0865974Z 
2022-12-01T10:49:12.0866056Z Generating XML reports...
2022-12-01T10:49:12.0866471Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201104646.xml
2022-12-01T10:49:12.0866742Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0866866Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0867158Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0867297Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0867483Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps3fteyh6
2022-12-01T10:49:12.0867681Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps3fteyh6/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0867949Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T10:49:12.0867954Z 
2022-12-01T10:49:12.0868027Z Running tests...
2022-12-01T10:49:12.0868222Z ----------------------------------------------------------------------
2022-12-01T10:49:12.0868501Z   test_device_maps_invalid_max_remote_device (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10192
2022-12-01T10:49:12.0880584Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10193
2022-12-01T10:49:12.0880855Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10194
2022-12-01T10:49:12.0881018Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10195
2022-12-01T10:49:12.0881342Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0881470Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0881756Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0881893Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0882082Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwrva65sk
2022-12-01T10:49:12.0882283Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwrva65sk/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0882559Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0882783Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0883103Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0883225Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0883409Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0rs3iyk7
2022-12-01T10:49:12.0883606Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0rs3iyk7/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0883881Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0884004Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0884286Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0884424Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0884612Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl4vf63fe
2022-12-01T10:49:12.0884811Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl4vf63fe/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0885070Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0885195Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0885476Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0885612Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0885798Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgh39dyjc
2022-12-01T10:49:12.0885995Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgh39dyjc/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0886163Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:49:12.0886329Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:49:12.0886480Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:49:12.0886643Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:49:12.0886744Z skip: Need at least 1 CUDA device (1.956s)
2022-12-01T10:49:12.0886751Z 
2022-12-01T10:49:12.0886952Z ----------------------------------------------------------------------
2022-12-01T10:49:12.0887029Z Ran 1 test in 1.956s
2022-12-01T10:49:12.0887035Z 
2022-12-01T10:49:12.0887107Z OK (skipped=1)
2022-12-01T10:49:12.0887112Z 
2022-12-01T10:49:12.0887197Z Generating XML reports...
2022-12-01T10:49:12.0887625Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201104650.xml
2022-12-01T10:49:12.0887903Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0888016Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0888298Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0888435Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0888623Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp164spmva
2022-12-01T10:49:12.0888820Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp164spmva/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0889091Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T10:49:12.0889096Z 
2022-12-01T10:49:12.0889169Z Running tests...
2022-12-01T10:49:12.0889396Z ----------------------------------------------------------------------
2022-12-01T10:49:12.0889706Z   test_device_maps_invalid_min_device (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10247
2022-12-01T10:49:12.0889853Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10248
2022-12-01T10:49:12.0890008Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10249
2022-12-01T10:49:12.0890161Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10250
2022-12-01T10:49:12.0890438Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0890563Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0890842Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0890979Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0891167Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppnjq2t5j
2022-12-01T10:49:12.0891352Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppnjq2t5j/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0891626Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0891751Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0892027Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0892164Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0892463Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9hfb3zjm
2022-12-01T10:49:12.0892665Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9hfb3zjm/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0892940Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0893065Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0893333Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0893470Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0893656Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcpe992gu
2022-12-01T10:49:12.0893851Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcpe992gu/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0894124Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0894248Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0894531Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0894668Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0894838Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1wbtoe5c
2022-12-01T10:49:12.0895035Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1wbtoe5c/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0895200Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:49:12.0895364Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:49:12.0895526Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:49:12.0895687Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:49:12.0895826Z skip: Need at least 1 CUDA device (1.955s)
2022-12-01T10:49:12.0895833Z 
2022-12-01T10:49:12.0896065Z ----------------------------------------------------------------------
2022-12-01T10:49:12.0896141Z Ran 1 test in 1.955s
2022-12-01T10:49:12.0896146Z 
2022-12-01T10:49:12.0896204Z OK (skipped=1)
2022-12-01T10:49:12.0896210Z 
2022-12-01T10:49:12.0896294Z Generating XML reports...
2022-12-01T10:49:12.0896715Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201104653.xml
2022-12-01T10:49:12.0896989Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0897113Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0897394Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0897532Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0897720Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv8obvavu
2022-12-01T10:49:12.0897906Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv8obvavu/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0898173Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T10:49:12.0898178Z 
2022-12-01T10:49:12.0898252Z Running tests...
2022-12-01T10:49:12.0898449Z ----------------------------------------------------------------------
2022-12-01T10:49:12.0898720Z   test_device_maps_many_to_one (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10302
2022-12-01T10:49:12.0898876Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10303
2022-12-01T10:49:12.0899030Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10304
2022-12-01T10:49:12.0899184Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10305
2022-12-01T10:49:12.0899459Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0899569Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0899852Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0899987Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0900173Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3racvk3b
2022-12-01T10:49:12.0900502Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3racvk3b/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0900802Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0900929Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0901208Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0901346Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0901517Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp24q9z580
2022-12-01T10:49:12.0901709Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp24q9z580/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0901978Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0902103Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0902384Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0902519Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0902771Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmfozt605
2022-12-01T10:49:12.0903000Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmfozt605/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0903255Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0903378Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0903658Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0903794Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0903982Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdatb3w8_
2022-12-01T10:49:12.0904176Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdatb3w8_/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0904346Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:49:12.0904515Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:49:12.0904680Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:49:12.0904827Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:49:12.0904931Z skip: Need at least 2 CUDA devices (1.856s)
2022-12-01T10:49:12.0904937Z 
2022-12-01T10:49:12.0905135Z ----------------------------------------------------------------------
2022-12-01T10:49:12.0905212Z Ran 1 test in 1.857s
2022-12-01T10:49:12.0905218Z 
2022-12-01T10:49:12.0905290Z OK (skipped=1)
2022-12-01T10:49:12.0905295Z 
2022-12-01T10:49:12.0905380Z Generating XML reports...
2022-12-01T10:49:12.0905800Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201104657.xml
2022-12-01T10:49:12.0906077Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0906204Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0906473Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0906608Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0906794Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprlkqy1jq
2022-12-01T10:49:12.0906991Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprlkqy1jq/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0907257Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T10:49:12.0907263Z 
2022-12-01T10:49:12.0907335Z Running tests...
2022-12-01T10:49:12.0907533Z ----------------------------------------------------------------------
2022-12-01T10:49:12.0907808Z   test_device_maps_missing_config (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10357
2022-12-01T10:49:12.0907956Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10358
2022-12-01T10:49:12.0908112Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10359
2022-12-01T10:49:12.0908265Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10360
2022-12-01T10:49:12.0908537Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0908661Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0908940Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0909076Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0909291Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfyjwzolb
2022-12-01T10:49:12.0909520Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfyjwzolb/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0909781Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0909904Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0910180Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0910316Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0910502Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpoj25oe_x
2022-12-01T10:49:12.0910698Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpoj25oe_x/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0910969Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0911097Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0911363Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0911497Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0911683Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe2ox6lz2
2022-12-01T10:49:12.0911878Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe2ox6lz2/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0912145Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0912269Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0912550Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0912688Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0912872Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpirl8d0q2
2022-12-01T10:49:12.0913052Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpirl8d0q2/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0913215Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:49:12.0913378Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:49:12.0913540Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:49:12.0913701Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:49:12.0913802Z skip: Need at least 1 CUDA device (1.955s)
2022-12-01T10:49:12.0913807Z 
2022-12-01T10:49:12.0914007Z ----------------------------------------------------------------------
2022-12-01T10:49:12.0914087Z Ran 1 test in 1.955s
2022-12-01T10:49:12.0914092Z 
2022-12-01T10:49:12.0914164Z OK (skipped=1)
2022-12-01T10:49:12.0914169Z 
2022-12-01T10:49:12.0914240Z Generating XML reports...
2022-12-01T10:49:12.0914656Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201104700.xml
2022-12-01T10:49:12.0914927Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0915050Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0915328Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0915464Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0915677Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkpcoz8kl
2022-12-01T10:49:12.0915903Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkpcoz8kl/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0916158Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T10:49:12.0916178Z 
2022-12-01T10:49:12.0916237Z Running tests...
2022-12-01T10:49:12.0916434Z ----------------------------------------------------------------------
2022-12-01T10:49:12.0916712Z   test_device_maps_missing_config_loop (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10412
2022-12-01T10:49:12.0916869Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10413
2022-12-01T10:49:12.0917025Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10414
2022-12-01T10:49:12.0917179Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10415
2022-12-01T10:49:12.0917454Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0917582Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0917847Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0917985Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0918169Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdhrwmsxe
2022-12-01T10:49:12.0918369Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdhrwmsxe/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0918639Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0918767Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0919047Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0919185Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0919371Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsw8ptf1z
2022-12-01T10:49:12.0919555Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsw8ptf1z/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0919824Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0919947Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0920228Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0920363Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0920549Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxxgrl1c5
2022-12-01T10:49:12.0920744Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxxgrl1c5/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0921017Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0921128Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0921408Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0921542Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0921726Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdwzgq52m
2022-12-01T10:49:12.0921922Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdwzgq52m/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0922086Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:49:12.0922299Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:49:12.0922490Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:49:12.0922655Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:49:12.0922744Z skip: Need at least 1 CUDA device (1.855s)
2022-12-01T10:49:12.0922749Z 
2022-12-01T10:49:12.0922947Z ----------------------------------------------------------------------
2022-12-01T10:49:12.0923023Z Ran 1 test in 1.855s
2022-12-01T10:49:12.0923028Z 
2022-12-01T10:49:12.0923103Z OK (skipped=1)
2022-12-01T10:49:12.0923108Z 
2022-12-01T10:49:12.0923194Z Generating XML reports...
2022-12-01T10:49:12.0923609Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201104703.xml
2022-12-01T10:49:12.0923884Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0924013Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0924280Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0924418Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0924604Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8dtt72lr
2022-12-01T10:49:12.0924803Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8dtt72lr/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0925074Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T10:49:12.0925079Z 
2022-12-01T10:49:12.0925153Z Running tests...
2022-12-01T10:49:12.0925353Z ----------------------------------------------------------------------
2022-12-01T10:49:12.0925642Z   test_device_maps_missing_config_not_timeout (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10467
2022-12-01T10:49:12.0925806Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10468
2022-12-01T10:49:12.0925947Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10469
2022-12-01T10:49:12.0926103Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10470
2022-12-01T10:49:12.0926380Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0926507Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0926787Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0926924Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0927112Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2zl941xy
2022-12-01T10:49:12.0927313Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2zl941xy/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0927583Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0927693Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0927972Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0928108Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0928294Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp92pdl6ky
2022-12-01T10:49:12.0928489Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp92pdl6ky/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0928791Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0928946Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0929226Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0929347Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0929531Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp00y25h4c
2022-12-01T10:49:12.0929724Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp00y25h4c/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0929993Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0930117Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0930403Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0930539Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0930724Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpngt2x_1k
2022-12-01T10:49:12.0930916Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpngt2x_1k/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0931068Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:49:12.0931234Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:49:12.0931399Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:49:12.0931561Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:49:12.0931663Z skip: Need at least 1 CUDA device (1.857s)
2022-12-01T10:49:12.0931668Z 
2022-12-01T10:49:12.0931868Z ----------------------------------------------------------------------
2022-12-01T10:49:12.0931944Z Ran 1 test in 1.858s
2022-12-01T10:49:12.0931951Z 
2022-12-01T10:49:12.0932025Z OK (skipped=1)
2022-12-01T10:49:12.0932030Z 
2022-12-01T10:49:12.0932100Z Generating XML reports...
2022-12-01T10:49:12.0932619Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201104707.xml
2022-12-01T10:49:12.0932927Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0933052Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0933331Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0933467Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0933657Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw3t14lpt
2022-12-01T10:49:12.0933854Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw3t14lpt/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0934128Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T10:49:12.0934133Z 
2022-12-01T10:49:12.0934192Z Running tests...
2022-12-01T10:49:12.0934392Z ----------------------------------------------------------------------
2022-12-01T10:49:12.0934675Z   test_device_maps_missing_config_remote (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10522
2022-12-01T10:49:12.0934831Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10523
2022-12-01T10:49:12.0934986Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10524
2022-12-01T10:49:12.0935139Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10525
2022-12-01T10:49:12.0935451Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0935606Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0935887Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0936011Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0936196Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbbjhtyrx
2022-12-01T10:49:12.0936393Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbbjhtyrx/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0936663Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0936786Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0937063Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0937200Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0937386Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc5rp5l69
2022-12-01T10:49:12.0937566Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc5rp5l69/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0937835Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0937957Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0938232Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0938368Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0938552Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsuv5hg6k
2022-12-01T10:49:12.0938751Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsuv5hg6k/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0939020Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0939144Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0939412Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0939548Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0939731Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptvegor3e
2022-12-01T10:49:12.0939929Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptvegor3e/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0940095Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:49:12.0940261Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:49:12.0940555Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:49:12.0940719Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:49:12.0940807Z skip: Need at least 1 CUDA device (1.953s)
2022-12-01T10:49:12.0940826Z 
2022-12-01T10:49:12.0941014Z ----------------------------------------------------------------------
2022-12-01T10:49:12.0941126Z Ran 1 test in 1.954s
2022-12-01T10:49:12.0941131Z 
2022-12-01T10:49:12.0941205Z OK (skipped=1)
2022-12-01T10:49:12.0941209Z 
2022-12-01T10:49:12.0941296Z Generating XML reports...
2022-12-01T10:49:12.0941714Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201104710.xml
2022-12-01T10:49:12.0942042Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0942169Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0942487Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0942609Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0942797Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph12ctr9f
2022-12-01T10:49:12.0942994Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph12ctr9f/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0943263Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T10:49:12.0943268Z 
2022-12-01T10:49:12.0943342Z Running tests...
2022-12-01T10:49:12.0943539Z ----------------------------------------------------------------------
2022-12-01T10:49:12.0943835Z   test_device_maps_missing_config_remote_response (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10577
2022-12-01T10:49:12.0943997Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10578
2022-12-01T10:49:12.0944154Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10579
2022-12-01T10:49:12.0944295Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10580
2022-12-01T10:49:12.0944571Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0944696Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0944977Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0945114Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0945302Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5lqx3ndq
2022-12-01T10:49:12.0945500Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5lqx3ndq/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0945771Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0945894Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0946159Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0946294Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0946478Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq32rr5iz
2022-12-01T10:49:12.0946673Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq32rr5iz/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0946943Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0947067Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0947351Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0947485Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0947656Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzdti7fxo
2022-12-01T10:49:12.0947855Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzdti7fxo/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0948125Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0948249Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0948527Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0948693Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0948907Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxriqtx5u
2022-12-01T10:49:12.0949107Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxriqtx5u/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0949272Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:49:12.0949422Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:49:12.0949586Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:49:12.0949746Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:49:12.0949849Z skip: Need at least 1 CUDA device (1.954s)
2022-12-01T10:49:12.0949854Z 
2022-12-01T10:49:12.0950051Z ----------------------------------------------------------------------
2022-12-01T10:49:12.0950128Z Ran 1 test in 1.954s
2022-12-01T10:49:12.0950133Z 
2022-12-01T10:49:12.0950212Z OK (skipped=1)
2022-12-01T10:49:12.0950217Z 
2022-12-01T10:49:12.0950302Z Generating XML reports...
2022-12-01T10:49:12.0950704Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201104714.xml
2022-12-01T10:49:12.0950976Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0951102Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0951379Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0951515Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0951702Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbjged7_g
2022-12-01T10:49:12.0951901Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbjged7_g/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0952172Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T10:49:12.0952177Z 
2022-12-01T10:49:12.0952251Z Running tests...
2022-12-01T10:49:12.0952434Z ----------------------------------------------------------------------
2022-12-01T10:49:12.0952716Z   test_device_maps_missing_config_response (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10632
2022-12-01T10:49:12.0952874Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10633
2022-12-01T10:49:12.0953028Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10634
2022-12-01T10:49:12.0953180Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10635
2022-12-01T10:49:12.0953454Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0953582Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0953862Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0953995Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0954167Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpepk9dzz4
2022-12-01T10:49:12.0954361Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpepk9dzz4/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0954633Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0954756Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0955065Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0955204Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0955417Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzab9mzvt
2022-12-01T10:49:12.0955613Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzab9mzvt/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0955870Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0955995Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0956274Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0956409Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0956593Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpud7z4nnc
2022-12-01T10:49:12.0956790Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpud7z4nnc/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0957063Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0957188Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0957469Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0957593Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0957778Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbawl4pdv
2022-12-01T10:49:12.0957975Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbawl4pdv/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0958141Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:49:12.0958311Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:49:12.0958473Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:49:12.0958636Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:49:12.0958737Z skip: Need at least 1 CUDA device (1.863s)
2022-12-01T10:49:12.0958742Z 
2022-12-01T10:49:12.0958941Z ----------------------------------------------------------------------
2022-12-01T10:49:12.0959004Z Ran 1 test in 1.864s
2022-12-01T10:49:12.0959009Z 
2022-12-01T10:49:12.0959080Z OK (skipped=1)
2022-12-01T10:49:12.0959085Z 
2022-12-01T10:49:12.0959172Z Generating XML reports...
2022-12-01T10:49:12.0959590Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201104717.xml
2022-12-01T10:49:12.0959868Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0959996Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0960277Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0960415Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0960587Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi6bwkbbm
2022-12-01T10:49:12.0960783Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi6bwkbbm/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0961050Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T10:49:12.0961055Z 
2022-12-01T10:49:12.0961130Z Running tests...
2022-12-01T10:49:12.0961327Z ----------------------------------------------------------------------
2022-12-01T10:49:12.0961638Z   test_device_maps_missing_config_response_loop (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10687
2022-12-01T10:49:12.0961827Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10688
2022-12-01T10:49:12.0961980Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10689
2022-12-01T10:49:12.0962134Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10690
2022-12-01T10:49:12.0962396Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0962520Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0962801Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0962936Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0963122Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg7uug5e5
2022-12-01T10:49:12.0963319Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg7uug5e5/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0963593Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0963718Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0963996Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0964118Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0964303Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsm_r9q02
2022-12-01T10:49:12.0964496Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsm_r9q02/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0964768Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0964895Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0965177Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0965311Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0965497Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppkwdh5qx
2022-12-01T10:49:12.0965680Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppkwdh5qx/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0965949Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0966073Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0966351Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0966488Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0966676Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjzrqjyme
2022-12-01T10:49:12.0966871Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjzrqjyme/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0967036Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:49:12.0967201Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:49:12.0967349Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:49:12.0967509Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:49:12.0967611Z skip: Need at least 1 CUDA device (1.956s)
2022-12-01T10:49:12.0967616Z 
2022-12-01T10:49:12.0967811Z ----------------------------------------------------------------------
2022-12-01T10:49:12.0967887Z Ran 1 test in 1.957s
2022-12-01T10:49:12.0967931Z 
2022-12-01T10:49:12.0968006Z OK (skipped=1)
2022-12-01T10:49:12.0968040Z 
2022-12-01T10:49:12.0968126Z Generating XML reports...
2022-12-01T10:49:12.0968540Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201104721.xml
2022-12-01T10:49:12.0968798Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0968924Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0969201Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0969340Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0969524Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1f82l4v2
2022-12-01T10:49:12.0969722Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1f82l4v2/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0969991Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T10:49:12.0969997Z 
2022-12-01T10:49:12.0970071Z Running tests...
2022-12-01T10:49:12.0970269Z ----------------------------------------------------------------------
2022-12-01T10:49:12.0970524Z   test_device_maps_multi_gpu (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10742
2022-12-01T10:49:12.0970682Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10743
2022-12-01T10:49:12.0970837Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10744
2022-12-01T10:49:12.0970991Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10745
2022-12-01T10:49:12.0971264Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0971388Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0971673Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0971809Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0971996Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqdsvzhk4
2022-12-01T10:49:12.0972256Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0972449Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0972736Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0972873Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0973058Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt8m73_m5
2022-12-01T10:49:12.0973258Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqdsvzhk4/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0973449Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt8m73_m5/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0973722Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0973831Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0974112Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0974247Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0974432Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkkb0fru1
2022-12-01T10:49:12.0974668Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkkb0fru1/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0974939Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0975096Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0975380Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0975519Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0975690Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplzu2jdma
2022-12-01T10:49:12.0975888Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplzu2jdma/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0976055Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:49:12.0976220Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:49:12.0976386Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:49:12.0976551Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:49:12.0976656Z skip: Need at least 2 CUDA devices (1.853s)
2022-12-01T10:49:12.0976662Z 
2022-12-01T10:49:12.0976859Z ----------------------------------------------------------------------
2022-12-01T10:49:12.0976921Z Ran 1 test in 1.853s
2022-12-01T10:49:12.0976940Z 
2022-12-01T10:49:12.0976999Z OK (skipped=1)
2022-12-01T10:49:12.0977004Z 
2022-12-01T10:49:12.0977088Z Generating XML reports...
2022-12-01T10:49:12.0977503Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201104724.xml
2022-12-01T10:49:12.0977776Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0977904Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0978183Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0978322Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0978507Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc3bob4w8
2022-12-01T10:49:12.0978689Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc3bob4w8/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0978956Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T10:49:12.0978961Z 
2022-12-01T10:49:12.0979035Z Running tests...
2022-12-01T10:49:12.0979231Z ----------------------------------------------------------------------
2022-12-01T10:49:12.0979504Z   test_device_maps_multi_gpu_self (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10797
2022-12-01T10:49:12.0979664Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10798
2022-12-01T10:49:12.0979822Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10799
2022-12-01T10:49:12.0979974Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10800
2022-12-01T10:49:12.0980246Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0980490Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0980787Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0980925Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0981111Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfa3f6eej
2022-12-01T10:49:12.0981361Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfa3f6eej/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0981671Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0981795Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0982072Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0982194Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0982379Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq4byem1p
2022-12-01T10:49:12.0982574Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq4byem1p/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0982846Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0982972Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0983252Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0983389Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0983578Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptbxujsyf
2022-12-01T10:49:12.0983772Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptbxujsyf/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0984031Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0984156Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0984435Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0984571Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0984757Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfzwrjgl5
2022-12-01T10:49:12.0984955Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfzwrjgl5/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0985121Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:49:12.0985287Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:49:12.0985436Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:49:12.0985599Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:49:12.0985707Z skip: Need at least 2 CUDA devices (1.856s)
2022-12-01T10:49:12.0985713Z 
2022-12-01T10:49:12.0985911Z ----------------------------------------------------------------------
2022-12-01T10:49:12.0985988Z Ran 1 test in 1.856s
2022-12-01T10:49:12.0985993Z 
2022-12-01T10:49:12.0986066Z OK (skipped=1)
2022-12-01T10:49:12.0986072Z 
2022-12-01T10:49:12.0986161Z Generating XML reports...
2022-12-01T10:49:12.0986580Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201104728.xml
2022-12-01T10:49:12.0986856Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0986967Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0987248Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0987386Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0987571Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9kum3rry
2022-12-01T10:49:12.0987768Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9kum3rry/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0988066Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T10:49:12.0988098Z 
2022-12-01T10:49:12.0988174Z Running tests...
2022-12-01T10:49:12.0988372Z ----------------------------------------------------------------------
2022-12-01T10:49:12.0988645Z   test_device_maps_one_to_many (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10852
2022-12-01T10:49:12.0988791Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10853
2022-12-01T10:49:12.0988947Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10854
2022-12-01T10:49:12.0989101Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10855
2022-12-01T10:49:12.0989378Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0989506Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0989787Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0989926Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0990113Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgoca4av5
2022-12-01T10:49:12.0990297Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgoca4av5/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0990573Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0990697Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0990978Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0991114Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0991301Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe39kynoi
2022-12-01T10:49:12.0991501Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe39kynoi/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0991772Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0991895Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0992162Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0992297Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0992481Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp85dow5kx
2022-12-01T10:49:12.0992677Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp85dow5kx/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0992949Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0993075Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0993353Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0993488Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0993673Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprru40wk_
2022-12-01T10:49:12.0993851Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprru40wk_/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0994016Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:49:12.0994180Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:49:12.0994373Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:49:12.0994536Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:49:12.0994671Z skip: Need at least 2 CUDA devices (1.854s)
2022-12-01T10:49:12.0994676Z 
2022-12-01T10:49:12.0994872Z ----------------------------------------------------------------------
2022-12-01T10:49:12.0994948Z Ran 1 test in 1.854s
2022-12-01T10:49:12.0994953Z 
2022-12-01T10:49:12.0995011Z OK (skipped=1)
2022-12-01T10:49:12.0995016Z 
2022-12-01T10:49:12.0995102Z Generating XML reports...
2022-12-01T10:49:12.0995517Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201104731.xml
2022-12-01T10:49:12.0995790Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0995914Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0996194Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0996334Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0996517Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7gfzlqdm
2022-12-01T10:49:12.0996713Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7gfzlqdm/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0996969Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T10:49:12.0996975Z 
2022-12-01T10:49:12.0997049Z Running tests...
2022-12-01T10:49:12.0997244Z ----------------------------------------------------------------------
2022-12-01T10:49:12.0997508Z   test_device_maps_remote (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10907
2022-12-01T10:49:12.0997670Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10908
2022-12-01T10:49:12.0997827Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10909
2022-12-01T10:49:12.0997982Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10910
2022-12-01T10:49:12.0998255Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0998366Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0998647Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0998787Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.0998974Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprpg752jm
2022-12-01T10:49:12.0999170Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprpg752jm/_remote_module_non_scriptable.py
2022-12-01T10:49:12.0999445Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.0999573Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.0999850Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.0999986Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.1000156Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxj38wk1z
2022-12-01T10:49:12.1000353Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxj38wk1z/_remote_module_non_scriptable.py
2022-12-01T10:49:12.1000627Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.1000749Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.1001069Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.1001232Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.1001418Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr92_ll0o
2022-12-01T10:49:12.1001688Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.1001811Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.1002078Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.1002213Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.1002399Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3ismafyc
2022-12-01T10:49:12.1002596Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr92_ll0o/_remote_module_non_scriptable.py
2022-12-01T10:49:12.1002791Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3ismafyc/_remote_module_non_scriptable.py
2022-12-01T10:49:12.1002956Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:49:12.1003119Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:49:12.1003281Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:49:12.1003429Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:49:12.1003534Z skip: Need at least 2 CUDA devices (1.955s)
2022-12-01T10:49:12.1003539Z 
2022-12-01T10:49:12.1003734Z ----------------------------------------------------------------------
2022-12-01T10:49:12.1003813Z Ran 1 test in 1.956s
2022-12-01T10:49:12.1003818Z 
2022-12-01T10:49:12.1003890Z OK (skipped=1)
2022-12-01T10:49:12.1003894Z 
2022-12-01T10:49:12.1003980Z Generating XML reports...
2022-12-01T10:49:12.1004397Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201104734.xml
2022-12-01T10:49:12.1004673Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.1004798Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.1005066Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.1005202Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.1005390Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfm1y2ujb
2022-12-01T10:49:12.1005589Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfm1y2ujb/_remote_module_non_scriptable.py
2022-12-01T10:49:12.1005862Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T10:49:12.1005869Z 
2022-12-01T10:49:12.1005942Z Running tests...
2022-12-01T10:49:12.1006141Z ----------------------------------------------------------------------
2022-12-01T10:49:12.1006412Z   test_device_maps_return_to_gpu (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10962
2022-12-01T10:49:12.1006573Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10963
2022-12-01T10:49:12.1006716Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10964
2022-12-01T10:49:12.1006870Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10965
2022-12-01T10:49:12.1007142Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.1007267Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.1007574Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.1007737Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.1007926Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxjptmao5
2022-12-01T10:49:12.1008202Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.1008312Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.1008593Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.1008728Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.1008913Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm2hi48ca
2022-12-01T10:49:12.1009114Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxjptmao5/_remote_module_non_scriptable.py
2022-12-01T10:49:12.1009306Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm2hi48ca/_remote_module_non_scriptable.py
2022-12-01T10:49:12.1009579Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.1009703Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.1009980Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.1010101Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.1010284Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4twca93o
2022-12-01T10:49:12.1010478Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4twca93o/_remote_module_non_scriptable.py
2022-12-01T10:49:12.1010748Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.1010875Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.1011153Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.1011287Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.1011471Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp51a4mx1i
2022-12-01T10:49:12.1011649Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp51a4mx1i/_remote_module_non_scriptable.py
2022-12-01T10:49:12.1011814Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:49:12.1011977Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:49:12.1012139Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:49:12.1012391Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:49:12.1012501Z skip: Need at least 4 CUDA devices (1.860s)
2022-12-01T10:49:12.1012507Z 
2022-12-01T10:49:12.1012708Z ----------------------------------------------------------------------
2022-12-01T10:49:12.1012784Z Ran 1 test in 1.861s
2022-12-01T10:49:12.1012788Z 
2022-12-01T10:49:12.1012861Z OK (skipped=1)
2022-12-01T10:49:12.1012865Z 
2022-12-01T10:49:12.1012936Z Generating XML reports...
2022-12-01T10:49:12.1013352Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201104738.xml
2022-12-01T10:49:12.1013627Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.1013750Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.1014087Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.1014248Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.1014433Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzwwlq03i
2022-12-01T10:49:12.1014627Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzwwlq03i/_remote_module_non_scriptable.py
2022-12-01T10:49:12.1014896Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T10:49:12.1014901Z 
2022-12-01T10:49:12.1014959Z Running tests...
2022-12-01T10:49:12.1015156Z ----------------------------------------------------------------------
2022-12-01T10:49:12.1015432Z   test_device_maps_return_to_gpu_self (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11017
2022-12-01T10:49:12.1015592Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11018
2022-12-01T10:49:12.1015749Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11019
2022-12-01T10:49:12.1015905Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11020
2022-12-01T10:49:12.1016182Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.1016305Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.1016570Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.1016710Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.1016895Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfpjhr483
2022-12-01T10:49:12.1017092Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfpjhr483/_remote_module_non_scriptable.py
2022-12-01T10:49:12.1017368Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.1017494Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.1017773Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.1017909Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.1018095Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps06lqzbj
2022-12-01T10:49:12.1018275Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps06lqzbj/_remote_module_non_scriptable.py
2022-12-01T10:49:12.1018545Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.1018669Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.1018954Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.1019089Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.1019276Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1x_30g66
2022-12-01T10:49:12.1019468Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1x_30g66/_remote_module_non_scriptable.py
2022-12-01T10:49:12.1019738Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.1019848Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.1020129Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.1020266Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.1020616Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp215ysx60
2022-12-01T10:49:12.1020871Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp215ysx60/_remote_module_non_scriptable.py
2022-12-01T10:49:12.1021069Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:49:12.1021233Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:49:12.1021394Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:49:12.1021556Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:49:12.1021645Z skip: Need at least 4 CUDA devices (1.856s)
2022-12-01T10:49:12.1021651Z 
2022-12-01T10:49:12.1021852Z ----------------------------------------------------------------------
2022-12-01T10:49:12.1021928Z Ran 1 test in 1.856s
2022-12-01T10:49:12.1021933Z 
2022-12-01T10:49:12.1022005Z OK (skipped=1)
2022-12-01T10:49:12.1022010Z 
2022-12-01T10:49:12.1022094Z Generating XML reports...
2022-12-01T10:49:12.1022514Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201104741.xml
2022-12-01T10:49:12.1022795Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.1022920Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.1023202Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.1023325Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.1023512Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1dbuf2t3
2022-12-01T10:49:12.1023707Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1dbuf2t3/_remote_module_non_scriptable.py
2022-12-01T10:49:12.1023978Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T10:49:12.1023985Z 
2022-12-01T10:49:12.1024059Z Running tests...
2022-12-01T10:49:12.1024262Z ----------------------------------------------------------------------
2022-12-01T10:49:12.1024538Z   test_device_maps_wrong_worker_name (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11072
2022-12-01T10:49:12.1024697Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11073
2022-12-01T10:49:12.1024841Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11074
2022-12-01T10:49:12.1024995Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11075
2022-12-01T10:49:12.1025268Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.1025392Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.1025675Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.1025814Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.1025999Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8io10t1q
2022-12-01T10:49:12.1026193Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8io10t1q/_remote_module_non_scriptable.py
2022-12-01T10:49:12.1026465Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.1026576Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.1026855Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.1026990Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.1027176Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphzwpuqj9
2022-12-01T10:49:12.1027407Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphzwpuqj9/_remote_module_non_scriptable.py
2022-12-01T10:49:12.1027719Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.1027843Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.1028121Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.1028258Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.1028429Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0nmsx7ig
2022-12-01T10:49:12.1028624Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0nmsx7ig/_remote_module_non_scriptable.py
2022-12-01T10:49:12.1028894Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.1029022Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.1029307Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.1029444Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.1029630Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1one6eiy
2022-12-01T10:49:12.1029823Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1one6eiy/_remote_module_non_scriptable.py
2022-12-01T10:49:12.1029975Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:49:12.1030141Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:49:12.1030305Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:49:12.1030468Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:49:12.1030576Z skip: Need at least 2 CUDA devices (1.854s)
2022-12-01T10:49:12.1030583Z 
2022-12-01T10:49:12.1030781Z ----------------------------------------------------------------------
2022-12-01T10:49:12.1030857Z Ran 1 test in 1.854s
2022-12-01T10:49:12.1030862Z 
2022-12-01T10:49:12.1030937Z OK (skipped=1)
2022-12-01T10:49:12.1030942Z 
2022-12-01T10:49:12.1031027Z Generating XML reports...
2022-12-01T10:49:12.1031430Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201104745.xml
2022-12-01T10:49:12.1031706Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.1031828Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.1032107Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.1032245Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.1032431Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkbqkb6vj
2022-12-01T10:49:12.1032626Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkbqkb6vj/_remote_module_non_scriptable.py
2022-12-01T10:49:12.1032896Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T10:49:12.1032902Z 
2022-12-01T10:49:12.1032976Z Running tests...
2022-12-01T10:49:12.1033160Z ----------------------------------------------------------------------
2022-12-01T10:49:12.1033424Z   test_device_mismatch (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11127
2022-12-01T10:49:12.1033579Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11128
2022-12-01T10:49:12.1033762Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11129
2022-12-01T10:49:12.1033920Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11130
2022-12-01T10:49:12.1034226Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.1034352Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.1034631Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.1034753Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.1034941Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplysx331g
2022-12-01T10:49:12.1035137Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplysx331g/_remote_module_non_scriptable.py
2022-12-01T10:49:12.1035408Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.1035535Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.1035815Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.1035951Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.1036137Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8bna8f0s
2022-12-01T10:49:12.1036330Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8bna8f0s/_remote_module_non_scriptable.py
2022-12-01T10:49:12.1036589Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.1036712Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.1036988Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.1037127Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.1037315Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphjinfatl
2022-12-01T10:49:12.1037512Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphjinfatl/_remote_module_non_scriptable.py
2022-12-01T10:49:12.1037782Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.1037906Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.1038171Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.1038307Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.1038491Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpohii601q
2022-12-01T10:49:12.1038687Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpohii601q/_remote_module_non_scriptable.py
2022-12-01T10:49:12.1038852Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:49:12.1039019Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:49:12.1039183Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:49:12.1039345Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:49:12.1039446Z skip: Need at least 1 CUDA device (1.854s)
2022-12-01T10:49:12.1039451Z 
2022-12-01T10:49:12.1039635Z ----------------------------------------------------------------------
2022-12-01T10:49:12.1039710Z Ran 1 test in 1.854s
2022-12-01T10:49:12.1039715Z 
2022-12-01T10:49:12.1039787Z OK (skipped=1)
2022-12-01T10:49:12.1039792Z 
2022-12-01T10:49:12.1039877Z Generating XML reports...
2022-12-01T10:49:12.1040324Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201104748.xml
2022-12-01T10:49:12.1040626Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.1040752Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.1041035Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.1041158Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.1041342Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp80jggdch
2022-12-01T10:49:12.1041538Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp80jggdch/_remote_module_non_scriptable.py
2022-12-01T10:49:12.1041805Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T10:49:12.1041810Z 
2022-12-01T10:49:12.1041884Z Running tests...
2022-12-01T10:49:12.1042083Z ----------------------------------------------------------------------
2022-12-01T10:49:12.1042355Z   test_devices_option_mismatch (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11182
2022-12-01T10:49:12.1042512Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11183
2022-12-01T10:49:12.1042667Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11184
2022-12-01T10:49:12.1042807Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11185
2022-12-01T10:49:12.1043084Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.1043209Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.1043489Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.1043628Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.1043816Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdtwifi5c
2022-12-01T10:49:12.1044089Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.1044213Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.1044496Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.1044618Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.1044804Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9irjsmgc
2022-12-01T10:49:12.1045002Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdtwifi5c/_remote_module_non_scriptable.py
2022-12-01T10:49:12.1045196Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9irjsmgc/_remote_module_non_scriptable.py
2022-12-01T10:49:12.1045472Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.1045594Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.1045873Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.1046010Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.1046181Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6ljzcaiy
2022-12-01T10:49:12.1046376Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6ljzcaiy/_remote_module_non_scriptable.py
2022-12-01T10:49:12.1046652Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.1046778Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.1047089Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.1047252Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.1047439Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyb84iko_
2022-12-01T10:49:12.1047634Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyb84iko_/_remote_module_non_scriptable.py
2022-12-01T10:49:12.1047801Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:49:12.1047953Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:49:12.1048116Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:49:12.1048279Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:49:12.1048386Z skip: Need at least 2 CUDA devices (1.854s)
2022-12-01T10:49:12.1048392Z 
2022-12-01T10:49:12.1048592Z ----------------------------------------------------------------------
2022-12-01T10:49:12.1048670Z Ran 1 test in 1.855s
2022-12-01T10:49:12.1048675Z 
2022-12-01T10:49:12.1048747Z OK (skipped=1)
2022-12-01T10:49:12.1048752Z 
2022-12-01T10:49:12.1048837Z Generating XML reports...
2022-12-01T10:49:12.1049242Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201104751.xml
2022-12-01T10:49:12.1049519Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.1049646Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.1049933Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.1050072Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.1050258Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8ar7ectv
2022-12-01T10:49:12.1050457Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8ar7ectv/_remote_module_non_scriptable.py
2022-12-01T10:49:12.1050725Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T10:49:12.1050730Z 
2022-12-01T10:49:12.1050804Z Running tests...
2022-12-01T10:49:12.1050990Z ----------------------------------------------------------------------
2022-12-01T10:49:12.1051271Z   test_devices_option_mismatch_reverse (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11237
2022-12-01T10:49:12.1051430Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11238
2022-12-01T10:49:12.1051585Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11239
2022-12-01T10:49:12.1051740Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11240
2022-12-01T10:49:12.1052017Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.1052141Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.1052495Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.1052632Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.1052805Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw771dp47
2022-12-01T10:49:12.1053001Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw771dp47/_remote_module_non_scriptable.py
2022-12-01T10:49:12.1053277Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.1053440Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.1053717Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.1053884Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.1054069Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj55bjdyi
2022-12-01T10:49:12.1054344Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.1054454Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.1054738Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.1054876Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.1055060Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbdgde6x8
2022-12-01T10:49:12.1055258Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj55bjdyi/_remote_module_non_scriptable.py
2022-12-01T10:49:12.1055453Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbdgde6x8/_remote_module_non_scriptable.py
2022-12-01T10:49:12.1055723Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.1055847Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.1056127Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.1056247Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.1056432Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpquik6h4f
2022-12-01T10:49:12.1056624Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpquik6h4f/_remote_module_non_scriptable.py
2022-12-01T10:49:12.1056792Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:49:12.1056959Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:49:12.1057121Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:49:12.1057285Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:49:12.1057389Z skip: Need at least 2 CUDA devices (1.855s)
2022-12-01T10:49:12.1057394Z 
2022-12-01T10:49:12.1057594Z ----------------------------------------------------------------------
2022-12-01T10:49:12.1057657Z Ran 1 test in 1.855s
2022-12-01T10:49:12.1057662Z 
2022-12-01T10:49:12.1057736Z OK (skipped=1)
2022-12-01T10:49:12.1057740Z 
2022-12-01T10:49:12.1057825Z Generating XML reports...
2022-12-01T10:49:12.1058245Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201104755.xml
2022-12-01T10:49:12.1058519Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.1058644Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.1058925Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.1059060Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.1059231Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmru085tc
2022-12-01T10:49:12.1059423Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmru085tc/_remote_module_non_scriptable.py
2022-12-01T10:49:12.1059691Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T10:49:12.1059696Z 
2022-12-01T10:49:12.1059770Z Running tests...
2022-12-01T10:49:12.1060020Z ----------------------------------------------------------------------
2022-12-01T10:49:12.1060302Z   test_owner_rref_forward_synchronization1 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11292
2022-12-01T10:49:12.1060595Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11293
2022-12-01T10:49:12.1060751Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11294
2022-12-01T10:49:12.1060904Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11295
2022-12-01T10:49:12.1061166Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.1061290Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.1061572Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.1061711Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.1061896Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp85z30le3
2022-12-01T10:49:12.1062094Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp85z30le3/_remote_module_non_scriptable.py
2022-12-01T10:49:12.1062364Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.1062488Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.1062767Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.1062888Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.1063072Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprz_7vwbj
2022-12-01T10:49:12.1063344Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.1063467Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.1063747Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.1063882Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.1064066Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp88v88n20
2022-12-01T10:49:12.1064260Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprz_7vwbj/_remote_module_non_scriptable.py
2022-12-01T10:49:12.1064435Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp88v88n20/_remote_module_non_scriptable.py
2022-12-01T10:49:12.1064706Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.1064831Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.1065111Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.1065253Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.1065436Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmdt2zus2
2022-12-01T10:49:12.1065629Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmdt2zus2/_remote_module_non_scriptable.py
2022-12-01T10:49:12.1065792Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:49:12.1065957Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:49:12.1066106Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:49:12.1066269Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:49:12.1066371Z skip: Need at least 1 CUDA device (1.854s)
2022-12-01T10:49:12.1066428Z 
2022-12-01T10:49:12.1066624Z ----------------------------------------------------------------------
2022-12-01T10:49:12.1066737Z Ran 1 test in 1.854s
2022-12-01T10:49:12.1066742Z 
2022-12-01T10:49:12.1066816Z OK (skipped=1)
2022-12-01T10:49:12.1066820Z 
2022-12-01T10:49:12.1066904Z Generating XML reports...
2022-12-01T10:49:12.1067320Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201104758.xml
2022-12-01T10:49:12.1067581Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.1067706Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.1067983Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.1068118Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.1068306Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsmu34k70
2022-12-01T10:49:12.1068503Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsmu34k70/_remote_module_non_scriptable.py
2022-12-01T10:49:12.1068773Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T10:49:12.1068778Z 
2022-12-01T10:49:12.1068852Z Running tests...
2022-12-01T10:49:12.1069049Z ----------------------------------------------------------------------
2022-12-01T10:49:12.1069319Z   test_owner_rref_forward_synchronization2 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11347
2022-12-01T10:49:12.1069477Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11348
2022-12-01T10:49:12.1069633Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11349
2022-12-01T10:49:12.1069787Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11350
2022-12-01T10:49:12.1070059Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.1070187Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.1070469Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.1070608Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.1070792Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3mnw0pc1
2022-12-01T10:49:12.1070974Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3mnw0pc1/_remote_module_non_scriptable.py
2022-12-01T10:49:12.1071247Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.1071371Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.1071652Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.1071790Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.1071976Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphfvq8amz
2022-12-01T10:49:12.1072172Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphfvq8amz/_remote_module_non_scriptable.py
2022-12-01T10:49:12.1072439Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.1072550Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.1072831Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.1072965Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.1073182Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3rtlg3zz
2022-12-01T10:49:12.1073408Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3rtlg3zz/_remote_module_non_scriptable.py
2022-12-01T10:49:12.1073678Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.1073802Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.1074081Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.1074217Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.1074388Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6yr1nv8q
2022-12-01T10:49:12.1074579Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6yr1nv8q/_remote_module_non_scriptable.py
2022-12-01T10:49:12.1074744Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:49:12.1074911Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:49:12.1075075Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:49:12.1075236Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:49:12.1075342Z skip: Need at least 2 CUDA devices (1.860s)
2022-12-01T10:49:12.1075347Z 
2022-12-01T10:49:12.1075545Z ----------------------------------------------------------------------
2022-12-01T10:49:12.1075607Z Ran 1 test in 1.860s
2022-12-01T10:49:12.1075625Z 
2022-12-01T10:49:12.1075682Z OK (skipped=1)
2022-12-01T10:49:12.1075687Z 
2022-12-01T10:49:12.1075771Z Generating XML reports...
2022-12-01T10:49:12.1076186Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201104802.xml
2022-12-01T10:49:12.1076463Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.1076588Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.1076872Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.1077007Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.1077193Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq4852nea
2022-12-01T10:49:12.1077377Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq4852nea/_remote_module_non_scriptable.py
2022-12-01T10:49:12.1077647Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T10:49:12.1077652Z 
2022-12-01T10:49:12.1077723Z Running tests...
2022-12-01T10:49:12.1077920Z ----------------------------------------------------------------------
2022-12-01T10:49:12.1078208Z   test_owner_rref_forward_synchronization3 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11402
2022-12-01T10:49:12.1078367Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11403
2022-12-01T10:49:12.1078524Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11404
2022-12-01T10:49:12.1078678Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11405
2022-12-01T10:49:12.1078951Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.1079063Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.1079347Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.1079483Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.1079698Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnl1jdhvx
2022-12-01T10:49:12.1079924Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnl1jdhvx/_remote_module_non_scriptable.py
2022-12-01T10:49:12.1080199Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.1080324Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.1080604Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.1080726Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.1080910Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7s64xv3a
2022-12-01T10:49:12.1081105Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7s64xv3a/_remote_module_non_scriptable.py
2022-12-01T10:49:12.1081378Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.1081504Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.1081784Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.1081918Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.1082103Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9ydlirs1
2022-12-01T10:49:12.1082298Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9ydlirs1/_remote_module_non_scriptable.py
2022-12-01T10:49:12.1082555Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.1082680Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.1082960Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.1083098Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.1083282Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps86ja_je
2022-12-01T10:49:12.1083473Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps86ja_je/_remote_module_non_scriptable.py
2022-12-01T10:49:12.1083640Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:49:12.1083804Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:49:12.1083968Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:49:12.1084117Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:49:12.1084222Z skip: Need at least 2 CUDA devices (1.953s)
2022-12-01T10:49:12.1084227Z 
2022-12-01T10:49:12.1084424Z ----------------------------------------------------------------------
2022-12-01T10:49:12.1084503Z Ran 1 test in 1.953s
2022-12-01T10:49:12.1084509Z 
2022-12-01T10:49:12.1084580Z OK (skipped=1)
2022-12-01T10:49:12.1084585Z 
2022-12-01T10:49:12.1084669Z Generating XML reports...
2022-12-01T10:49:12.1085084Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201104805.xml
2022-12-01T10:49:12.1085360Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.1085470Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.1085751Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.1085887Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.1086104Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptvpgxpj2
2022-12-01T10:49:12.1086329Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptvpgxpj2/_remote_module_non_scriptable.py
2022-12-01T10:49:12.1086599Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T10:49:12.1086604Z 
2022-12-01T10:49:12.1086678Z Running tests...
2022-12-01T10:49:12.1086873Z ----------------------------------------------------------------------
2022-12-01T10:49:12.1087157Z   test_owner_rref_forward_synchronization4 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11457
2022-12-01T10:49:12.1087302Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11458
2022-12-01T10:49:12.1087458Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11459
2022-12-01T10:49:12.1087609Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11460
2022-12-01T10:49:12.1087884Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.1088009Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.1088289Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.1088424Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.1088610Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph61ewtep
2022-12-01T10:49:12.1088806Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph61ewtep/_remote_module_non_scriptable.py
2022-12-01T10:49:12.1089064Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.1089189Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.1089469Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.1089606Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.1089789Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxgi8nk_v
2022-12-01T10:49:12.1089985Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxgi8nk_v/_remote_module_non_scriptable.py
2022-12-01T10:49:12.1090252Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.1090376Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.1090637Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.1090773Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.1090961Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvvbyq69p
2022-12-01T10:49:12.1091159Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvvbyq69p/_remote_module_non_scriptable.py
2022-12-01T10:49:12.1091431Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.1091555Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.1091834Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.1091968Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.1092154Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnp09_lh6
2022-12-01T10:49:12.1092400Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnp09_lh6/_remote_module_non_scriptable.py
2022-12-01T10:49:12.1092571Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:49:12.1092771Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:49:12.1092993Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:49:12.1093154Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:49:12.1093259Z skip: Need at least 2 CUDA devices (1.955s)
2022-12-01T10:49:12.1093265Z 
2022-12-01T10:49:12.1093466Z ----------------------------------------------------------------------
2022-12-01T10:49:12.1093543Z Ran 1 test in 1.955s
2022-12-01T10:49:12.1093548Z 
2022-12-01T10:49:12.1093606Z OK (skipped=1)
2022-12-01T10:49:12.1093624Z 
2022-12-01T10:49:12.1093694Z Generating XML reports...
2022-12-01T10:49:12.1094112Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201104809.xml
2022-12-01T10:49:12.1094391Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.1094518Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.1094800Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.1094936Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.1095120Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp06rj1c_7
2022-12-01T10:49:12.1095314Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp06rj1c_7/_remote_module_non_scriptable.py
2022-12-01T10:49:12.1095568Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T10:49:12.1095587Z 
2022-12-01T10:49:12.1095646Z Running tests...
2022-12-01T10:49:12.1095842Z ----------------------------------------------------------------------
2022-12-01T10:49:12.1096124Z   test_rref_as_arg_synchronization1 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11512
2022-12-01T10:49:12.1096284Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11513
2022-12-01T10:49:12.1096441Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11514
2022-12-01T10:49:12.1096595Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11515
2022-12-01T10:49:12.1096870Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.1096995Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.1097264Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.1097400Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.1097589Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgttef9g_
2022-12-01T10:49:12.1097787Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgttef9g_/_remote_module_non_scriptable.py
2022-12-01T10:49:12.1098059Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.1098184Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.1098462Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.1098597Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.1098769Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxwuep25l
2022-12-01T10:49:12.1098966Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxwuep25l/_remote_module_non_scriptable.py
2022-12-01T10:49:12.1099264Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.1099415Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.1099697Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.1099832Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.1100014Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_x7aj13x
2022-12-01T10:49:12.1100206Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_x7aj13x/_remote_module_non_scriptable.py
2022-12-01T10:49:12.1100589Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.1100699Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.1100982Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.1101115Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.1101301Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvxgm_f0x
2022-12-01T10:49:12.1101497Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvxgm_f0x/_remote_module_non_scriptable.py
2022-12-01T10:49:12.1101666Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:49:12.1101830Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:49:12.1101995Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:49:12.1102141Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:49:12.1102245Z skip: Need at least 1 CUDA device (1.954s)
2022-12-01T10:49:12.1102251Z 
2022-12-01T10:49:12.1102449Z ----------------------------------------------------------------------
2022-12-01T10:49:12.1102525Z Ran 1 test in 1.954s
2022-12-01T10:49:12.1102532Z 
2022-12-01T10:49:12.1102606Z OK (skipped=1)
2022-12-01T10:49:12.1102611Z 
2022-12-01T10:49:12.1102740Z Generating XML reports...
2022-12-01T10:49:12.1103163Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201104812.xml
2022-12-01T10:49:12.1103438Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.1103563Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.1103828Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.1103963Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.1104147Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg4ewrtln
2022-12-01T10:49:12.1104343Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg4ewrtln/_remote_module_non_scriptable.py
2022-12-01T10:49:12.1104615Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T10:49:12.1104620Z 
2022-12-01T10:49:12.1104693Z Running tests...
2022-12-01T10:49:12.1104890Z ----------------------------------------------------------------------
2022-12-01T10:49:12.1105164Z   test_rref_as_arg_synchronization2 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11567
2022-12-01T10:49:12.1105325Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11568
2022-12-01T10:49:12.1105467Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11569
2022-12-01T10:49:12.1105621Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11570
2022-12-01T10:49:12.1105957Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.1106115Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.1106399Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.1106534Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.1106720Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5topprkb
2022-12-01T10:49:12.1106919Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5topprkb/_remote_module_non_scriptable.py
2022-12-01T10:49:12.1107178Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.1107302Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.1107582Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.1107718Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.1107908Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1su1zp_r
2022-12-01T10:49:12.1108106Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1su1zp_r/_remote_module_non_scriptable.py
2022-12-01T10:49:12.1108378Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.1108501Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.1108783Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.1108903Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.1109087Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp266aqnt0
2022-12-01T10:49:12.1109284Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp266aqnt0/_remote_module_non_scriptable.py
2022-12-01T10:49:12.1109556Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.1109681Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.1109960Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.1110096Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.1110281Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzg6pg0v6
2022-12-01T10:49:12.1110474Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzg6pg0v6/_remote_module_non_scriptable.py
2022-12-01T10:49:12.1110625Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:49:12.1110791Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:49:12.1110955Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:49:12.1111119Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:49:12.1111224Z skip: Need at least 2 CUDA devices (1.953s)
2022-12-01T10:49:12.1111230Z 
2022-12-01T10:49:12.1111427Z ----------------------------------------------------------------------
2022-12-01T10:49:12.1111504Z Ran 1 test in 1.953s
2022-12-01T10:49:12.1111509Z 
2022-12-01T10:49:12.1111581Z OK (skipped=1)
2022-12-01T10:49:12.1111586Z 
2022-12-01T10:49:12.1111657Z Generating XML reports...
2022-12-01T10:49:12.1112074Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201104816.xml
2022-12-01T10:49:12.1112380Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.1112507Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.1112818Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.1112956Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.1113144Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz0n3d7fg
2022-12-01T10:49:12.1113341Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz0n3d7fg/_remote_module_non_scriptable.py
2022-12-01T10:49:12.1113609Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T10:49:12.1113615Z 
2022-12-01T10:49:12.1113674Z Running tests...
2022-12-01T10:49:12.1113872Z ----------------------------------------------------------------------
2022-12-01T10:49:12.1114627Z   test_rref_as_arg_synchronization3 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/81962 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.236s)
2022-12-01T10:49:12.1114635Z 
2022-12-01T10:49:12.1114829Z ----------------------------------------------------------------------
2022-12-01T10:49:12.1114907Z Ran 1 test in 0.236s
2022-12-01T10:49:12.1114912Z 
2022-12-01T10:49:12.1114984Z OK (skipped=1)
2022-12-01T10:49:12.1114988Z 
2022-12-01T10:49:12.1115071Z Generating XML reports...
2022-12-01T10:49:12.1115488Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201104819.xml
2022-12-01T10:49:12.1115762Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.1115888Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.1116155Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.1116291Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.1116477Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyho00vh3
2022-12-01T10:49:12.1116672Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyho00vh3/_remote_module_non_scriptable.py
2022-12-01T10:49:12.1116941Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T10:49:12.1116945Z 
2022-12-01T10:49:12.1117019Z Running tests...
2022-12-01T10:49:12.1117214Z ----------------------------------------------------------------------
2022-12-01T10:49:12.1117490Z   test_rref_as_arg_synchronization4 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11632
2022-12-01T10:49:12.1117648Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11633
2022-12-01T10:49:12.1117791Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11634
2022-12-01T10:49:12.1117944Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11635
2022-12-01T10:49:12.1118217Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.1118340Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.1118620Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.1118755Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.1118942Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi5lwubuu
2022-12-01T10:49:12.1119172Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi5lwubuu/_remote_module_non_scriptable.py
2022-12-01T10:49:12.1119458Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.1119583Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.1119864Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.1119998Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.1120184Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr7p4oo5r
2022-12-01T10:49:12.1120378Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr7p4oo5r/_remote_module_non_scriptable.py
2022-12-01T10:49:12.1120650Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.1120776Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.1121052Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.1121176Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.1121361Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuz91ia0f
2022-12-01T10:49:12.1121555Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuz91ia0f/_remote_module_non_scriptable.py
2022-12-01T10:49:12.1121826Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.1121951Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.1122229Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.1122361Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.1122549Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyj458i43
2022-12-01T10:49:12.1122729Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyj458i43/_remote_module_non_scriptable.py
2022-12-01T10:49:12.1122932Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:49:12.1123094Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:49:12.1123256Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:49:12.1123416Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:49:12.1123524Z skip: Need at least 2 CUDA devices (1.855s)
2022-12-01T10:49:12.1123529Z 
2022-12-01T10:49:12.1123727Z ----------------------------------------------------------------------
2022-12-01T10:49:12.1123803Z Ran 1 test in 1.855s
2022-12-01T10:49:12.1123808Z 
2022-12-01T10:49:12.1123883Z OK (skipped=1)
2022-12-01T10:49:12.1123888Z 
2022-12-01T10:49:12.1123963Z Generating XML reports...
2022-12-01T10:49:12.1124380Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201104821.xml
2022-12-01T10:49:12.1124653Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.1124776Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.1125058Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.1125193Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.1125376Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkao7zr9b
2022-12-01T10:49:12.1125600Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkao7zr9b/_remote_module_non_scriptable.py
2022-12-01T10:49:12.1125870Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T10:49:12.1125926Z 
2022-12-01T10:49:12.1125986Z Running tests...
2022-12-01T10:49:12.1126183Z ----------------------------------------------------------------------
2022-12-01T10:49:12.1126459Z   test_rref_as_arg_synchronization5 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11687
2022-12-01T10:49:12.1126617Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11688
2022-12-01T10:49:12.1126772Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11689
2022-12-01T10:49:12.1126924Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11690
2022-12-01T10:49:12.1127200Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.1127327Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.1127595Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.1127732Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.1127918Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpaqf5d70x
2022-12-01T10:49:12.1128115Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpaqf5d70x/_remote_module_non_scriptable.py
2022-12-01T10:49:12.1128390Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.1128514Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.1128792Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.1128930Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.1129115Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpaj55phdh
2022-12-01T10:49:12.1129301Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpaj55phdh/_remote_module_non_scriptable.py
2022-12-01T10:49:12.1129573Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.1129699Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.1129978Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.1130111Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.1130296Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplywsd_u0
2022-12-01T10:49:12.1130489Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplywsd_u0/_remote_module_non_scriptable.py
2022-12-01T10:49:12.1130764Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.1130891Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.1131155Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.1131294Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.1131477Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptz2x_s75
2022-12-01T10:49:12.1131669Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptz2x_s75/_remote_module_non_scriptable.py
2022-12-01T10:49:12.1131834Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:49:12.1131998Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:49:12.1132187Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:49:12.1132455Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:49:12.1132547Z skip: Need at least 1 CUDA device (1.956s)
2022-12-01T10:49:12.1132553Z 
2022-12-01T10:49:12.1132755Z ----------------------------------------------------------------------
2022-12-01T10:49:12.1132830Z Ran 1 test in 1.956s
2022-12-01T10:49:12.1132835Z 
2022-12-01T10:49:12.1132909Z OK (skipped=1)
2022-12-01T10:49:12.1132914Z 
2022-12-01T10:49:12.1132997Z Generating XML reports...
2022-12-01T10:49:12.1133413Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201104824.xml
2022-12-01T10:49:12.1133687Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.1133814Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.1134094Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.1134219Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.1134407Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpavkd54hg
2022-12-01T10:49:12.1134603Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpavkd54hg/_remote_module_non_scriptable.py
2022-12-01T10:49:12.1134870Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T10:49:12.1134875Z 
2022-12-01T10:49:12.1134950Z Running tests...
2022-12-01T10:49:12.1135146Z ----------------------------------------------------------------------
2022-12-01T10:49:12.1135425Z   test_rref_forward_synchronization1 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11742
2022-12-01T10:49:12.1135587Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11743
2022-12-01T10:49:12.1135744Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11744
2022-12-01T10:49:12.1135883Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11745
2022-12-01T10:49:12.1136158Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.1136281Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.1136560Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.1136695Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.1136880Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyyqwckrb
2022-12-01T10:49:12.1137080Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyyqwckrb/_remote_module_non_scriptable.py
2022-12-01T10:49:12.1137352Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.1137465Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.1137747Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.1137883Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.1138068Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv2l02n4c
2022-12-01T10:49:12.1138262Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv2l02n4c/_remote_module_non_scriptable.py
2022-12-01T10:49:12.1138533Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.1138655Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.1138971Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.1139136Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.1139306Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7hg4de75
2022-12-01T10:49:12.1139498Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7hg4de75/_remote_module_non_scriptable.py
2022-12-01T10:49:12.1139768Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.1139889Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.1140165Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.1140298Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.1140595Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6v35lh7c
2022-12-01T10:49:12.1140788Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6v35lh7c/_remote_module_non_scriptable.py
2022-12-01T10:49:12.1140939Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:49:12.1141106Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:49:12.1141271Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:49:12.1141432Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:49:12.1141534Z skip: Need at least 1 CUDA device (1.855s)
2022-12-01T10:49:12.1141539Z 
2022-12-01T10:49:12.1141737Z ----------------------------------------------------------------------
2022-12-01T10:49:12.1141814Z Ran 1 test in 1.855s
2022-12-01T10:49:12.1141819Z 
2022-12-01T10:49:12.1141891Z OK (skipped=1)
2022-12-01T10:49:12.1141896Z 
2022-12-01T10:49:12.1141984Z Generating XML reports...
2022-12-01T10:49:12.1142386Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201104828.xml
2022-12-01T10:49:12.1142658Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.1142781Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.1143062Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.1143196Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.1143381Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpiaxe7652
2022-12-01T10:49:12.1143580Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpiaxe7652/_remote_module_non_scriptable.py
2022-12-01T10:49:12.1143849Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T10:49:12.1143856Z 
2022-12-01T10:49:12.1143930Z Running tests...
2022-12-01T10:49:12.1144113Z ----------------------------------------------------------------------
2022-12-01T10:49:12.1144394Z   test_rref_forward_synchronization2 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11797
2022-12-01T10:49:12.1144551Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11798
2022-12-01T10:49:12.1144706Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11799
2022-12-01T10:49:12.1144861Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11800
2022-12-01T10:49:12.1145135Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.1145258Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.1145587Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.1145742Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.1145926Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjgo4v6sf
2022-12-01T10:49:12.1146139Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjgo4v6sf/_remote_module_non_scriptable.py
2022-12-01T10:49:12.1146412Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.1146538Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.1146816Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.1146954Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.1147140Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3zr35o0o
2022-12-01T10:49:12.1147412Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.1147522Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.1147801Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.1147934Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.1148118Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpko7xh3r2
2022-12-01T10:49:12.1148389Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.1148513Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.1148795Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.1148932Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.1149101Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0_l5hkoa
2022-12-01T10:49:12.1149298Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3zr35o0o/_remote_module_non_scriptable.py
2022-12-01T10:49:12.1149488Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpko7xh3r2/_remote_module_non_scriptable.py
2022-12-01T10:49:12.1149683Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0_l5hkoa/_remote_module_non_scriptable.py
2022-12-01T10:49:12.1149848Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:49:12.1150012Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:49:12.1150176Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:49:12.1150341Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:49:12.1150448Z skip: Need at least 2 CUDA devices (1.852s)
2022-12-01T10:49:12.1150453Z 
2022-12-01T10:49:12.1150637Z ----------------------------------------------------------------------
2022-12-01T10:49:12.1150713Z Ran 1 test in 1.853s
2022-12-01T10:49:12.1150718Z 
2022-12-01T10:49:12.1150790Z OK (skipped=1)
2022-12-01T10:49:12.1150795Z 
2022-12-01T10:49:12.1150881Z Generating XML reports...
2022-12-01T10:49:12.1151304Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201104831.xml
2022-12-01T10:49:12.1151576Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.1151704Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.1152025Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.1152188Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.1152358Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe0omz8ag
2022-12-01T10:49:12.1152552Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe0omz8ag/_remote_module_non_scriptable.py
2022-12-01T10:49:12.1152820Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T10:49:12.1152825Z 
2022-12-01T10:49:12.1152899Z Running tests...
2022-12-01T10:49:12.1153096Z ----------------------------------------------------------------------
2022-12-01T10:49:12.1153372Z   test_rref_forward_synchronization3 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11852
2022-12-01T10:49:12.1153534Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11853
2022-12-01T10:49:12.1153691Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11854
2022-12-01T10:49:12.1153833Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11855
2022-12-01T10:49:12.1154108Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.1154232Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.1154515Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.1154652Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.1154838Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj7_nax1x
2022-12-01T10:49:12.1155033Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj7_nax1x/_remote_module_non_scriptable.py
2022-12-01T10:49:12.1155308Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.1155660Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.1156092Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.1156440Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.1156790Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp339ji4ad
2022-12-01T10:49:12.1157180Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp339ji4ad/_remote_module_non_scriptable.py
2022-12-01T10:49:12.1157668Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.1157998Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.1158412Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.1158760Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.1159101Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl_r2sd8r
2022-12-01T10:49:12.1159501Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl_r2sd8r/_remote_module_non_scriptable.py
2022-12-01T10:49:12.1159976Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.1160309Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.1160736Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.1161083Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.1161418Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj7gczxul
2022-12-01T10:49:12.1161859Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj7gczxul/_remote_module_non_scriptable.py
2022-12-01T10:49:12.1162266Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:49:12.1162605Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:49:12.1162956Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:49:12.1163302Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:49:12.1163589Z skip: Need at least 2 CUDA devices (1.851s)
2022-12-01T10:49:12.1163725Z 
2022-12-01T10:49:12.1163911Z ----------------------------------------------------------------------
2022-12-01T10:49:12.1164155Z Ran 1 test in 1.852s
2022-12-01T10:49:12.1164268Z 
2022-12-01T10:49:12.1164343Z OK (skipped=1)
2022-12-01T10:49:12.1164455Z 
2022-12-01T10:49:12.1164538Z Generating XML reports...
2022-12-01T10:49:12.1165063Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201104834.xml
2022-12-01T10:49:12.1165669Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.1165998Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.1166416Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.1166760Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.1167101Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2wi6d7mi
2022-12-01T10:49:12.1167533Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2wi6d7mi/_remote_module_non_scriptable.py
2022-12-01T10:49:12.1168015Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T10:49:12.1168234Z 
2022-12-01T10:49:12.1168317Z Running tests...
2022-12-01T10:49:12.1168623Z ----------------------------------------------------------------------
2022-12-01T10:49:12.1169069Z   test_rref_forward_synchronization4 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11907
2022-12-01T10:49:12.1169499Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11908
2022-12-01T10:49:12.1169834Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11909
2022-12-01T10:49:12.1170166Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11910
2022-12-01T10:49:12.1170618Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.1170938Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.1171375Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.1171727Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.1172076Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq0q76x13
2022-12-01T10:49:12.1172562Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq0q76x13/_remote_module_non_scriptable.py
2022-12-01T10:49:12.1173054Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.1173382Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.1173808Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.1174138Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.1174538Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4qk0cl46
2022-12-01T10:49:12.1174941Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4qk0cl46/_remote_module_non_scriptable.py
2022-12-01T10:49:12.1175457Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.1175770Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.1176196Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.1176543Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.1176877Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr97whz8s
2022-12-01T10:49:12.1177275Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr97whz8s/_remote_module_non_scriptable.py
2022-12-01T10:49:12.1177760Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.1178093Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.1178504Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.1178851Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.1179192Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphp408a8v
2022-12-01T10:49:12.1179575Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphp408a8v/_remote_module_non_scriptable.py
2022-12-01T10:49:12.1179954Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:49:12.1180300Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:49:12.1180851Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:49:12.1181185Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:49:12.1181472Z skip: Need at least 2 CUDA devices (1.855s)
2022-12-01T10:49:12.1181609Z 
2022-12-01T10:49:12.1181809Z ----------------------------------------------------------------------
2022-12-01T10:49:12.1182037Z Ran 1 test in 1.855s
2022-12-01T10:49:12.1182151Z 
2022-12-01T10:49:12.1182222Z OK (skipped=1)
2022-12-01T10:49:12.1182329Z 
2022-12-01T10:49:12.1182414Z Generating XML reports...
2022-12-01T10:49:12.1182949Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201104838.xml
2022-12-01T10:49:12.1183539Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.1183872Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.1184302Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.1184642Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.1184974Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp39r68a8g
2022-12-01T10:49:12.1185372Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp39r68a8g/_remote_module_non_scriptable.py
2022-12-01T10:49:12.1185866Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T10:49:12.1186087Z 
2022-12-01T10:49:12.1186146Z Running tests...
2022-12-01T10:49:12.1186448Z ----------------------------------------------------------------------
2022-12-01T10:49:12.1186889Z   test_rref_to_here_synchronization1 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11962
2022-12-01T10:49:12.1187334Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11963
2022-12-01T10:49:12.1187710Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11964
2022-12-01T10:49:12.1188082Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11965
2022-12-01T10:49:12.1188546Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.1188905Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.1189338Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.1189670Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.1190013Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplctrcjry
2022-12-01T10:49:12.1190415Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplctrcjry/_remote_module_non_scriptable.py
2022-12-01T10:49:12.1190889Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.1191219Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.1191645Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.1191994Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.1192324Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0yc12xyj
2022-12-01T10:49:12.1192720Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0yc12xyj/_remote_module_non_scriptable.py
2022-12-01T10:49:12.1193203Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.1193515Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.1193946Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.1194290Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.1194638Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwh79lqqa
2022-12-01T10:49:12.1195022Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwh79lqqa/_remote_module_non_scriptable.py
2022-12-01T10:49:12.1195542Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.1195872Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.1196295Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.1196624Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.1196966Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxfvluk4i
2022-12-01T10:49:12.1197366Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxfvluk4i/_remote_module_non_scriptable.py
2022-12-01T10:49:12.1197732Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:49:12.1198079Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:49:12.1198423Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:49:12.1198768Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:49:12.1199041Z skip: Need at least 1 CUDA device (1.954s)
2022-12-01T10:49:12.1199177Z 
2022-12-01T10:49:12.1199377Z ----------------------------------------------------------------------
2022-12-01T10:49:12.1199619Z Ran 1 test in 1.954s
2022-12-01T10:49:12.1199731Z 
2022-12-01T10:49:12.1199788Z OK (skipped=1)
2022-12-01T10:49:12.1199895Z 
2022-12-01T10:49:12.1199980Z Generating XML reports...
2022-12-01T10:49:12.1200562Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201104841.xml
2022-12-01T10:49:12.1202638Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.1202981Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.1203410Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.1203755Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.1204095Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0fttv0e1
2022-12-01T10:49:12.1204495Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0fttv0e1/_remote_module_non_scriptable.py
2022-12-01T10:49:12.1204959Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T10:49:12.1205180Z 
2022-12-01T10:49:12.1205257Z Running tests...
2022-12-01T10:49:12.1205559Z ----------------------------------------------------------------------
2022-12-01T10:49:12.1205988Z   test_rref_to_here_synchronization2 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12017
2022-12-01T10:49:12.1206427Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12018
2022-12-01T10:49:12.1206757Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12019
2022-12-01T10:49:12.1207085Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 12020
2022-12-01T10:49:12.1207523Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.1207849Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.1208278Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.1208610Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.1208956Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0_zufq7f
2022-12-01T10:49:12.1209350Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0_zufq7f/_remote_module_non_scriptable.py
2022-12-01T10:49:12.1209830Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.1210145Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.1210570Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.1210916Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.1211257Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjv_gpaat
2022-12-01T10:49:12.1211647Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjv_gpaat/_remote_module_non_scriptable.py
2022-12-01T10:49:12.1212153Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.1212598Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.1213029Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.1213368Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.1213708Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcuq1vj4o
2022-12-01T10:49:12.1214127Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcuq1vj4o/_remote_module_non_scriptable.py
2022-12-01T10:49:12.1214618Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.1214990Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.1215453Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.1215801Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.1216162Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdxxk9o6i
2022-12-01T10:49:12.1216556Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdxxk9o6i/_remote_module_non_scriptable.py
2022-12-01T10:49:12.1216944Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:49:12.1217292Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:49:12.1217642Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:49:12.1217977Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:49:12.1218263Z skip: Need at least 2 CUDA devices (1.953s)
2022-12-01T10:49:12.1218405Z 
2022-12-01T10:49:12.1218608Z ----------------------------------------------------------------------
2022-12-01T10:49:12.1218837Z Ran 1 test in 1.953s
2022-12-01T10:49:12.1218946Z 
2022-12-01T10:49:12.1219018Z OK (skipped=1)
2022-12-01T10:49:12.1219123Z 
2022-12-01T10:49:12.1219209Z Generating XML reports...
2022-12-01T10:49:12.1219963Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201104845.xml
2022-12-01T10:49:12.1221006Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.1221606Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.1222148Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.1222487Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.1222838Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp7msbj82
2022-12-01T10:49:12.1223245Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp7msbj82/_remote_module_non_scriptable.py
2022-12-01T10:49:12.1223733Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T10:49:12.1223953Z 
2022-12-01T10:49:12.1224013Z Running tests...
2022-12-01T10:49:12.1224315Z ----------------------------------------------------------------------
2022-12-01T10:49:12.1224762Z   test_rref_to_here_synchronization3 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12072
2022-12-01T10:49:12.1225199Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12073
2022-12-01T10:49:12.1225522Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12074
2022-12-01T10:49:12.1225870Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 12075
2022-12-01T10:49:12.1226323Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.1226638Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.1227066Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.1227412Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.1227757Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd72dx_wz
2022-12-01T10:49:12.1228141Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd72dx_wz/_remote_module_non_scriptable.py
2022-12-01T10:49:12.1228625Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.1229096Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.1229554Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.1229905Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.1230248Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvgut7d19
2022-12-01T10:49:12.1230734Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvgut7d19/_remote_module_non_scriptable.py
2022-12-01T10:49:12.1231284Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.1231614Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.1232038Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.1232387Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.1232721Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwt9rm6sp
2022-12-01T10:49:12.1233120Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwt9rm6sp/_remote_module_non_scriptable.py
2022-12-01T10:49:12.1233601Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.1233913Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.1234337Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.1234679Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.1235018Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc7ociyd_
2022-12-01T10:49:12.1235414Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc7ociyd_/_remote_module_non_scriptable.py
2022-12-01T10:49:12.1235792Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:49:12.1236140Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:49:12.1236484Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:49:12.1236815Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:49:12.1237096Z skip: Need at least 2 CUDA devices (1.956s)
2022-12-01T10:49:12.1237234Z 
2022-12-01T10:49:12.1237439Z ----------------------------------------------------------------------
2022-12-01T10:49:12.1237664Z Ran 1 test in 1.957s
2022-12-01T10:49:12.1237777Z 
2022-12-01T10:49:12.1237848Z OK (skipped=1)
2022-12-01T10:49:12.1237955Z 
2022-12-01T10:49:12.1238037Z Generating XML reports...
2022-12-01T10:49:12.1238559Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201104848.xml
2022-12-01T10:49:12.1239158Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.1239487Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.1239911Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.1240246Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.1240585Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwd1_zrx5
2022-12-01T10:49:12.1240979Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwd1_zrx5/_remote_module_non_scriptable.py
2022-12-01T10:49:12.1241456Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T10:49:12.1241659Z 
2022-12-01T10:49:12.1241731Z Running tests...
2022-12-01T10:49:12.1242076Z ----------------------------------------------------------------------
2022-12-01T10:49:12.1242550Z   test_rref_to_here_synchronization4 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12127
2022-12-01T10:49:12.1242994Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12128
2022-12-01T10:49:12.1243315Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12129
2022-12-01T10:49:12.1243643Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 12130
2022-12-01T10:49:12.1244096Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.1244416Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.1244847Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.1245198Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.1245545Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt31j0gu5
2022-12-01T10:49:12.1245933Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt31j0gu5/_remote_module_non_scriptable.py
2022-12-01T10:49:12.1246413Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.1246737Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.1247152Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.1247496Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.1247834Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5xqjk1re
2022-12-01T10:49:12.1248233Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5xqjk1re/_remote_module_non_scriptable.py
2022-12-01T10:49:12.1248703Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.1249035Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.1249462Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.1249801Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.1250133Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb41dtopu
2022-12-01T10:49:12.1250534Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb41dtopu/_remote_module_non_scriptable.py
2022-12-01T10:49:12.1251033Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.1251487Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.1252150Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.1252717Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.1253193Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp320gdxzj
2022-12-01T10:49:12.1253870Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp320gdxzj/_remote_module_non_scriptable.py
2022-12-01T10:49:12.1254377Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:49:12.1254729Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:49:12.1255062Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:49:12.1255405Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:49:12.1255758Z skip: Need at least 2 CUDA devices (1.857s)
2022-12-01T10:49:12.1255900Z 
2022-12-01T10:49:12.1256154Z ----------------------------------------------------------------------
2022-12-01T10:49:12.1256390Z Ran 1 test in 1.857s
2022-12-01T10:49:12.1256505Z 
2022-12-01T10:49:12.1256576Z OK (skipped=1)
2022-12-01T10:49:12.1256683Z 
2022-12-01T10:49:12.1256771Z Generating XML reports...
2022-12-01T10:49:12.1257297Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201104852.xml
2022-12-01T10:49:12.1257897Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.1258228Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.1258654Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.1258989Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.1259331Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp85xis1ra
2022-12-01T10:49:12.1259734Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp85xis1ra/_remote_module_non_scriptable.py
2022-12-01T10:49:12.1260213Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T10:49:12.1260559Z 
2022-12-01T10:49:12.1260637Z Running tests...
2022-12-01T10:49:12.1260945Z ----------------------------------------------------------------------
2022-12-01T10:49:12.1261392Z   test_rref_with_unpickleable_attributes (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12182
2022-12-01T10:49:12.1261826Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12183
2022-12-01T10:49:12.1262164Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12184
2022-12-01T10:49:12.1262494Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 12185
2022-12-01T10:49:12.1262953Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.1263268Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.1263698Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.1264043Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.1264867Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzp83_k6z
2022-12-01T10:49:12.1265431Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzp83_k6z/_remote_module_non_scriptable.py
2022-12-01T10:49:12.1266179Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.1266692Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.1267309Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.1267699Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.1268095Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxiks1r52
2022-12-01T10:49:12.1268844Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxiks1r52/_remote_module_non_scriptable.py
2022-12-01T10:49:12.1269334Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.1269666Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.1270096Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.1270425Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.1270872Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgk86eplo
2022-12-01T10:49:12.1271316Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgk86eplo/_remote_module_non_scriptable.py
2022-12-01T10:49:12.1271808Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.1272126Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.1272557Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.1272901Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.1273246Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2mkgrdi9
2022-12-01T10:49:12.1273633Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2mkgrdi9/_remote_module_non_scriptable.py
2022-12-01T10:49:12.1274013Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:49:12.1274363Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:49:12.1274695Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:49:12.1275037Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:49:12.1275316Z skip: Need at least 1 CUDA device (1.857s)
2022-12-01T10:49:12.1275453Z 
2022-12-01T10:49:12.1275654Z ----------------------------------------------------------------------
2022-12-01T10:49:12.1275885Z Ran 1 test in 1.858s
2022-12-01T10:49:12.1275996Z 
2022-12-01T10:49:12.1276069Z OK (skipped=1)
2022-12-01T10:49:12.1276175Z 
2022-12-01T10:49:12.1276261Z Generating XML reports...
2022-12-01T10:49:12.1276785Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201104855.xml
2022-12-01T10:49:12.1277543Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.1277885Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.1278320Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.1278656Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.1279004Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3af7xeh9
2022-12-01T10:49:12.1279407Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3af7xeh9/_remote_module_non_scriptable.py
2022-12-01T10:49:12.1279879Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T10:49:12.1280100Z 
2022-12-01T10:49:12.1280174Z Running tests...
2022-12-01T10:49:12.1280481Z ----------------------------------------------------------------------
2022-12-01T10:49:12.1280919Z   test_tensor_view_as_return_value (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12237
2022-12-01T10:49:12.1281352Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12238
2022-12-01T10:49:12.1281685Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12239
2022-12-01T10:49:12.1282013Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 12240
2022-12-01T10:49:12.1282466Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.1282783Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.1283211Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.1283595Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.1283928Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprakxdxfi
2022-12-01T10:49:12.1284368Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprakxdxfi/_remote_module_non_scriptable.py
2022-12-01T10:49:12.1284855Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.1285187Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.1285603Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.1285945Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.1286287Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe6eh1ega
2022-12-01T10:49:12.1286674Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe6eh1ega/_remote_module_non_scriptable.py
2022-12-01T10:49:12.1287168Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.1287500Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.1287933Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.1288267Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.1288609Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptiu_ozdi
2022-12-01T10:49:12.1289011Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptiu_ozdi/_remote_module_non_scriptable.py
2022-12-01T10:49:12.1289498Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.1289816Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.1290246Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.1290594Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.1290923Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe5249inw
2022-12-01T10:49:12.1291321Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe5249inw/_remote_module_non_scriptable.py
2022-12-01T10:49:12.1292139Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:49:12.1292722Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:49:12.1293297Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:49:12.1293884Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:49:12.1294361Z skip: Need at least 1 CUDA device (1.955s)
2022-12-01T10:49:12.1294590Z 
2022-12-01T10:49:12.1294931Z ----------------------------------------------------------------------
2022-12-01T10:49:12.1295345Z Ran 1 test in 1.956s
2022-12-01T10:49:12.1295536Z 
2022-12-01T10:49:12.1295653Z OK (skipped=1)
2022-12-01T10:49:12.1295840Z 
2022-12-01T10:49:12.1295989Z Generating XML reports...
2022-12-01T10:49:12.1296983Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201104858.xml
2022-12-01T10:49:12.1298097Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.1298697Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.1299489Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.1300113Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.1300963Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpip6xcnzh
2022-12-01T10:49:12.1301765Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpip6xcnzh/_remote_module_non_scriptable.py
2022-12-01T10:49:12.1302657Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T10:49:12.1303057Z 
2022-12-01T10:49:12.1303185Z Running tests...
2022-12-01T10:49:12.1303726Z ----------------------------------------------------------------------
2022-12-01T10:49:12.1304540Z   test_device_maps_backward_pass (__main__.TensorPipeTensorPipeCudaDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12292
2022-12-01T10:49:12.1305340Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12293
2022-12-01T10:49:12.1305952Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12294
2022-12-01T10:49:12.1306565Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 12295
2022-12-01T10:49:12.1307375Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.1307977Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.1308754Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.1309379Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.1309982Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph2pu5f7g
2022-12-01T10:49:12.1310727Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph2pu5f7g/_remote_module_non_scriptable.py
2022-12-01T10:49:12.1311599Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.1312201Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.1312995Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.1313622Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.1314231Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpezotgy81
2022-12-01T10:49:12.1314966Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpezotgy81/_remote_module_non_scriptable.py
2022-12-01T10:49:12.1315839Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.1316419Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.1317205Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.1317835Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.1318469Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3lv9trr9
2022-12-01T10:49:12.1319348Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.1319937Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.1320722Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.1321342Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.1321957Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbm0kv14p
2022-12-01T10:49:12.1322690Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3lv9trr9/_remote_module_non_scriptable.py
2022-12-01T10:49:12.1323430Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbm0kv14p/_remote_module_non_scriptable.py
2022-12-01T10:49:12.1324177Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:49:12.1324817Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:49:12.1325500Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:49:12.1326137Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:49:12.1326643Z skip: Need at least 4 CUDA devices (1.957s)
2022-12-01T10:49:12.1326887Z 
2022-12-01T10:49:12.1327245Z ----------------------------------------------------------------------
2022-12-01T10:49:12.1327669Z Ran 1 test in 1.957s
2022-12-01T10:49:12.1327866Z 
2022-12-01T10:49:12.1327977Z OK (skipped=1)
2022-12-01T10:49:12.1328167Z 
2022-12-01T10:49:12.1328319Z Generating XML reports...
2022-12-01T10:49:12.1329334Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeCudaDistAutogradTest-20221201104902.xml
2022-12-01T10:49:12.1330481Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.1331070Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.1331863Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.1332555Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.1333173Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpel16wxtu
2022-12-01T10:49:12.1333903Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpel16wxtu/_remote_module_non_scriptable.py
2022-12-01T10:49:12.1334801Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T10:49:12.1335197Z 
2022-12-01T10:49:12.1335323Z Running tests...
2022-12-01T10:49:12.1335841Z ----------------------------------------------------------------------
2022-12-01T10:49:12.1336597Z   test_dist_autograd_sync_streams (__main__.TensorPipeTensorPipeCudaDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12347
2022-12-01T10:49:12.1337420Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12348
2022-12-01T10:49:12.1338028Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12349
2022-12-01T10:49:12.1338623Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 12350
2022-12-01T10:49:12.1339470Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.1340071Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.1340970Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.1341606Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.1342252Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6axwwpp4
2022-12-01T10:49:12.1343008Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6axwwpp4/_remote_module_non_scriptable.py
2022-12-01T10:49:12.1343897Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.1344502Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.1345293Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.1345922Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.1346535Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplipgevnm
2022-12-01T10:49:12.1347281Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplipgevnm/_remote_module_non_scriptable.py
2022-12-01T10:49:12.1348323Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.1348961Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.1349751Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.1350370Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.1350984Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7mxkxuls
2022-12-01T10:49:12.1351719Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7mxkxuls/_remote_module_non_scriptable.py
2022-12-01T10:49:12.1352619Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.1353215Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.1354000Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.1354636Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.1355265Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyx4ucr6x
2022-12-01T10:49:12.1356003Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyx4ucr6x/_remote_module_non_scriptable.py
2022-12-01T10:49:12.1356680Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:49:12.1357320Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:49:12.1357953Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:49:12.1358589Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:49:12.1359093Z skip: Need at least 4 CUDA devices (1.954s)
2022-12-01T10:49:12.1359334Z 
2022-12-01T10:49:12.1359697Z ----------------------------------------------------------------------
2022-12-01T10:49:12.1360130Z Ran 1 test in 1.954s
2022-12-01T10:49:12.1360324Z 
2022-12-01T10:49:12.1360434Z OK (skipped=1)
2022-12-01T10:49:12.1360623Z 
2022-12-01T10:49:12.1360772Z Generating XML reports...
2022-12-01T10:49:12.1361810Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeCudaDistAutogradTest-20221201104905.xml
2022-12-01T10:49:12.1362947Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.1363534Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.1364315Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.1364947Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.1365559Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1u5ipmfo
2022-12-01T10:49:12.1366304Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1u5ipmfo/_remote_module_non_scriptable.py
2022-12-01T10:49:12.1367190Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T10:49:12.1367588Z 
2022-12-01T10:49:12.1367716Z Running tests...
2022-12-01T10:49:12.1368241Z ----------------------------------------------------------------------
2022-12-01T10:49:12.1369064Z   test_gradients_synchronizations (__main__.TensorPipeTensorPipeCudaDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12402
2022-12-01T10:49:12.1369894Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12403
2022-12-01T10:49:12.1370491Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12404
2022-12-01T10:49:12.1371037Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 12405
2022-12-01T10:49:12.1371877Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.1372459Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.1373247Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.1373624Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.1373975Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr6z4lib6
2022-12-01T10:49:12.1374381Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr6z4lib6/_remote_module_non_scriptable.py
2022-12-01T10:49:12.1374861Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.1375194Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.1375628Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.1375962Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.1376309Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptkhtgi22
2022-12-01T10:49:12.1376712Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptkhtgi22/_remote_module_non_scriptable.py
2022-12-01T10:49:12.1377202Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.1377516Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.1377942Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.1378286Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.1378633Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt6wz2qzg
2022-12-01T10:49:12.1379031Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt6wz2qzg/_remote_module_non_scriptable.py
2022-12-01T10:49:12.1379519Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:12.1379849Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:12.1380262Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:12.1380762Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:12.1381104Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz36b_2ow
2022-12-01T10:49:12.1381503Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz36b_2ow/_remote_module_non_scriptable.py
2022-12-01T10:49:12.1381865Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:49:12.1382218Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:49:12.1382745Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:49:12.1383082Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:49:12.1383370Z skip: Need at least 4 CUDA devices (1.956s)
2022-12-01T10:49:12.1383509Z 
2022-12-01T10:49:12.1383719Z ----------------------------------------------------------------------
2022-12-01T10:49:12.1383969Z Ran 1 test in 1.956s
2022-12-01T10:49:12.1384069Z 
2022-12-01T10:49:12.1384142Z OK (skipped=1)
2022-12-01T10:49:12.1384254Z 
2022-12-01T10:49:12.1384341Z Generating XML reports...
2022-12-01T10:49:12.1384891Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeCudaDistAutogradTest-20221201104909.xml
2022-12-01T10:49:12.1385230Z 
2022-12-01T10:49:12.1385559Z ##[endgroup]
2022-12-01T10:49:12.1386144Z FINISHED PRINTING LOG FILE of distributed/rpc/cuda/test_tensorpipe_agent (/var/lib/jenkins/workspace/test/test-reports/distributed-rpc-cuda-test_tensorpipe_agent_jd46ql5s)
2022-12-01T10:49:12.1386469Z 
2022-12-01T10:49:12.1386693Z Running distributed/optim/test_zero_redundancy_optimizer ... [2022-12-01 10:49:11.929892]
2022-12-01T10:49:12.1387252Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/optim/test_zero_redundancy_optimizer.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-12-01 10:49:11.930352]
2022-12-01T10:49:46.2232952Z 
2022-12-01T10:49:46.2233647Z Expand the folded group to see the log file of distributed/optim/test_zero_redundancy_optimizer
2022-12-01T10:49:46.2234863Z ##[group]PRINTING LOG FILE of distributed/optim/test_zero_redundancy_optimizer (/var/lib/jenkins/workspace/test/test-reports/distributed-optim-test_zero_redundancy_optimizer_mur7lrv5)
2022-12-01T10:49:46.2235914Z Test results will be stored in test-reports/python-unittest/distributed.optim.test_zero_redundancy_optimizer
2022-12-01T10:49:46.2243598Z 
2022-12-01T10:49:46.2244196Z Running tests...
2022-12-01T10:49:46.2244829Z ----------------------------------------------------------------------
2022-12-01T10:49:46.2245399Z   test_add_param_group (__main__.TestZeroRedundancyOptimizerDistributed)
2022-12-01T10:49:46.2246829Z Check that ZeroRedundancyOptimizer properly handles adding a new ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/67287 for allplatform(s) . If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.235s)
2022-12-01T10:49:46.2247633Z   test_collect_shards (__main__.TestZeroRedundancyOptimizerDistributed)
2022-12-01T10:49:46.2248067Z Check the state consolidation mechanism and the state dict exposed ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12457
2022-12-01T10:49:46.2248483Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12458
2022-12-01T10:49:46.2248958Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:46.2249280Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:46.2249716Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:46.2250074Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:46.2250389Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:49:46.2250859Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:46.2251195Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:46.2251638Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:46.2251976Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:46.2252376Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:49:46.2252668Z skip: CUDA is not available. (1.416s)
2022-12-01T10:49:46.2253088Z   test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_False_gradient_as_bucket_view_False_static_graph_False_shard_buckets_False (__main__.TestZeroRedundancyOptimizerDistributed)
2022-12-01T10:49:46.2253584Z Check that overlapping DDP with ZeRO using the given method determined ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-12-01T10:49:46.2254087Z   test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_False_gradient_as_bucket_view_False_static_graph_False_shard_buckets_True (__main__.TestZeroRedundancyOptimizerDistributed)
2022-12-01T10:49:46.2254782Z Check that overlapping DDP with ZeRO using the given method determined ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-12-01T10:49:46.2255342Z   test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_False_gradient_as_bucket_view_False_static_graph_True_shard_buckets_False (__main__.TestZeroRedundancyOptimizerDistributed)
2022-12-01T10:49:46.2255818Z Check that overlapping DDP with ZeRO using the given method determined ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-12-01T10:49:46.2256314Z   test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_False_gradient_as_bucket_view_False_static_graph_True_shard_buckets_True (__main__.TestZeroRedundancyOptimizerDistributed)
2022-12-01T10:49:46.2256807Z Check that overlapping DDP with ZeRO using the given method determined ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-12-01T10:49:46.2257309Z   test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_False_gradient_as_bucket_view_True_static_graph_False_shard_buckets_False (__main__.TestZeroRedundancyOptimizerDistributed)
2022-12-01T10:49:46.2257792Z Check that overlapping DDP with ZeRO using the given method determined ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-12-01T10:49:46.2258294Z   test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_False_gradient_as_bucket_view_True_static_graph_False_shard_buckets_True (__main__.TestZeroRedundancyOptimizerDistributed)
2022-12-01T10:49:46.2258792Z Check that overlapping DDP with ZeRO using the given method determined ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-12-01T10:49:46.2259285Z   test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_False_gradient_as_bucket_view_True_static_graph_True_shard_buckets_False (__main__.TestZeroRedundancyOptimizerDistributed)
2022-12-01T10:49:46.2259774Z Check that overlapping DDP with ZeRO using the given method determined ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-12-01T10:49:46.2260259Z   test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_False_gradient_as_bucket_view_True_static_graph_True_shard_buckets_True (__main__.TestZeroRedundancyOptimizerDistributed)
2022-12-01T10:49:46.2261144Z Check that overlapping DDP with ZeRO using the given method determined ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-12-01T10:49:46.2261648Z   test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_True_gradient_as_bucket_view_False_static_graph_False_shard_buckets_False (__main__.TestZeroRedundancyOptimizerDistributed)
2022-12-01T10:49:46.2262140Z Check that overlapping DDP with ZeRO using the given method determined ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-12-01T10:49:46.2262688Z   test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_True_gradient_as_bucket_view_False_static_graph_False_shard_buckets_True (__main__.TestZeroRedundancyOptimizerDistributed)
2022-12-01T10:49:46.2263185Z Check that overlapping DDP with ZeRO using the given method determined ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-12-01T10:49:46.2263760Z   test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_True_gradient_as_bucket_view_False_static_graph_True_shard_buckets_False (__main__.TestZeroRedundancyOptimizerDistributed)
2022-12-01T10:49:46.2264317Z Check that overlapping DDP with ZeRO using the given method determined ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-12-01T10:49:46.2264797Z   test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_True_gradient_as_bucket_view_False_static_graph_True_shard_buckets_True (__main__.TestZeroRedundancyOptimizerDistributed)
2022-12-01T10:49:46.2265357Z Check that overlapping DDP with ZeRO using the given method determined ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-12-01T10:49:46.2265925Z   test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_True_gradient_as_bucket_view_True_static_graph_False_shard_buckets_False (__main__.TestZeroRedundancyOptimizerDistributed)
2022-12-01T10:49:46.2266504Z Check that overlapping DDP with ZeRO using the given method determined ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-12-01T10:49:46.2267065Z   test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_True_gradient_as_bucket_view_True_static_graph_False_shard_buckets_True (__main__.TestZeroRedundancyOptimizerDistributed)
2022-12-01T10:49:46.2267610Z Check that overlapping DDP with ZeRO using the given method determined ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-12-01T10:49:46.2268158Z   test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_True_gradient_as_bucket_view_True_static_graph_True_shard_buckets_False (__main__.TestZeroRedundancyOptimizerDistributed)
2022-12-01T10:49:46.2268650Z Check that overlapping DDP with ZeRO using the given method determined ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-12-01T10:49:46.2269208Z   test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_True_gradient_as_bucket_view_True_static_graph_True_shard_buckets_True (__main__.TestZeroRedundancyOptimizerDistributed)
2022-12-01T10:49:46.2269685Z Check that overlapping DDP with ZeRO using the given method determined ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-12-01T10:49:46.2270131Z   test_local_optimizer_parity_optimizer_class_str_AdamW_maximize_False (__main__.TestZeroRedundancyOptimizerDistributed)
2022-12-01T10:49:46.2270665Z When combined with DDP, check that a local optimizer gives the same ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12479
2022-12-01T10:49:46.2271071Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12480
2022-12-01T10:49:46.2271553Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:46.2271896Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:46.2272333Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:46.2272761Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:46.2273080Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:49:46.2273545Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:46.2273890Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:46.2274312Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:46.2274663Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:46.2274997Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:49:46.2275281Z skip: CUDA is not available. (1.413s)
2022-12-01T10:49:46.2275624Z   test_local_optimizer_parity_optimizer_class_str_AdamW_maximize_True (__main__.TestZeroRedundancyOptimizerDistributed)
2022-12-01T10:49:46.2276097Z When combined with DDP, check that a local optimizer gives the same ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12501
2022-12-01T10:49:46.2276571Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12502
2022-12-01T10:49:46.2277021Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:46.2277354Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:46.2277785Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:46.2278135Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:46.2278448Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:49:46.2278908Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:46.2279291Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:46.2279756Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:46.2280098Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:46.2280429Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:49:46.2280715Z skip: CUDA is not available. (1.413s)
2022-12-01T10:49:46.2281049Z   test_local_optimizer_parity_optimizer_class_str_Adam_maximize_False (__main__.TestZeroRedundancyOptimizerDistributed)
2022-12-01T10:49:46.2281520Z When combined with DDP, check that a local optimizer gives the same ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12523
2022-12-01T10:49:46.2281973Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12524
2022-12-01T10:49:46.2282437Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:46.2282761Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:46.2283193Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:46.2283607Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:46.2283920Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:49:46.2284386Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:46.2284719Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:46.2285154Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:46.2285492Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:46.2285819Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:49:46.2286108Z skip: CUDA is not available. (1.413s)
2022-12-01T10:49:46.2286456Z   test_local_optimizer_parity_optimizer_class_str_Adam_maximize_True (__main__.TestZeroRedundancyOptimizerDistributed)
2022-12-01T10:49:46.2286905Z When combined with DDP, check that a local optimizer gives the same ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12545
2022-12-01T10:49:46.2287306Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12546
2022-12-01T10:49:46.2287763Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:46.2288080Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:46.2288519Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:46.2288871Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:46.2289203Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:49:46.2289650Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:46.2289982Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:46.2290413Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:46.2290761Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:46.2291070Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:49:46.2291355Z skip: CUDA is not available. (1.414s)
2022-12-01T10:49:46.2291752Z   test_local_optimizer_parity_optimizer_class_str_SGD_maximize_False (__main__.TestZeroRedundancyOptimizerDistributed)
2022-12-01T10:49:46.2292287Z When combined with DDP, check that a local optimizer gives the same ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12567
2022-12-01T10:49:46.2292730Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12568
2022-12-01T10:49:46.2293200Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:46.2293536Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:46.2293950Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:46.2294299Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:46.2294625Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:49:46.2295077Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:46.2295416Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:46.2295904Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:46.2296255Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:46.2296564Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:49:46.2296900Z skip: CUDA is not available. (1.414s)
2022-12-01T10:49:46.2297260Z   test_local_optimizer_parity_optimizer_class_str_SGD_maximize_True (__main__.TestZeroRedundancyOptimizerDistributed)
2022-12-01T10:49:46.2297725Z When combined with DDP, check that a local optimizer gives the same ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12589
2022-12-01T10:49:46.2298110Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12590
2022-12-01T10:49:46.2298571Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:46.2298906Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:46.2299324Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:46.2299677Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:46.2300006Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:49:46.2300636Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:46.2300981Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:46.2301410Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:46.2301764Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:46.2302079Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:49:46.2302365Z skip: CUDA is not available. (1.414s)
2022-12-01T10:49:46.2302658Z   test_lr_scheduler (__main__.TestZeroRedundancyOptimizerDistributed)
2022-12-01T10:49:46.2303072Z Check that a normal PyTorch ``lr_scheduler`` is usable with ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12611
2022-12-01T10:49:46.2303447Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12612
2022-12-01T10:49:46.2303900Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:46.2304232Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:46.2304665Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:46.2305088Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:46.2305462Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:49:46.2305928Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:46.2306245Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:46.2306680Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:46.2307030Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:46.2307357Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:49:46.2307626Z skip: CUDA is not available. (1.412s)
2022-12-01T10:49:46.2307933Z   test_multiple_param_groups (__main__.TestZeroRedundancyOptimizerDistributed)
2022-12-01T10:49:46.2308382Z Check parity between constructing ZeRO with multiple parameter groups ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12633
2022-12-01T10:49:46.2308779Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12634
2022-12-01T10:49:46.2309235Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:46.2309569Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:46.2309998Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:46.2310332Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:46.2310662Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:49:46.2311119Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:46.2311453Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:46.2311874Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:46.2312225Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:46.2312557Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:49:46.2312827Z skip: CUDA is not available. (1.413s)
2022-12-01T10:49:46.2313139Z   test_nondefault_process_group (__main__.TestZeroRedundancyOptimizerDistributed)
2022-12-01T10:49:46.2313695Z Check that ZeroRedundancyOptimizer works with a non-default process ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12655
2022-12-01T10:49:46.2314114Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12656
2022-12-01T10:49:46.2314558Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:46.2314892Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:46.2315324Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:46.2315659Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:46.2315985Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:49:46.2316379Z INFO:torch.testing._internal.common_distributed:Skipping `test_nondefault_process_group()` since world size of 2 is less than 4
2022-12-01T10:49:46.2316878Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:46.2317194Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:46.2317663Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:46.2318013Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:46.2318375Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:49:46.2318754Z INFO:torch.testing._internal.common_distributed:Skipping `test_nondefault_process_group()` since world size of 2 is less than 4
2022-12-01T10:49:46.2319044Z ok (1.513s)
2022-12-01T10:49:46.2319305Z   test_sharding (__main__.TestZeroRedundancyOptimizerDistributed)
2022-12-01T10:49:46.2320180Z Check ZeroRedundancyOptimizer's parameter sharding at construction ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/67295 for allplatform(s) . If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.001s)
2022-12-01T10:49:46.2320805Z   test_step (__main__.TestZeroRedundancyOptimizerDistributed)
2022-12-01T10:49:46.2321240Z Check that ZeroRedundancyOptimizer properly exposes the ``step()`` ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12677
2022-12-01T10:49:46.2321651Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12678
2022-12-01T10:49:46.2322091Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:46.2322424Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:46.2322856Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:46.2323316Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:46.2323631Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:49:46.2324101Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:46.2324438Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:46.2324877Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:46.2325213Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:46.2325539Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:49:46.2325822Z skip: CUDA is not available. (1.412s)
2022-12-01T10:49:46.2326110Z   test_step_with_closure (__main__.TestZeroRedundancyOptimizerDistributed)
2022-12-01T10:49:46.2326539Z Check that ZeroRedundancyOptimizer properly exposes the ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12699
2022-12-01T10:49:46.2326940Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12700
2022-12-01T10:49:46.2327398Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:46.2327719Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:46.2328151Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:46.2328497Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:46.2328808Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:49:46.2329270Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:46.2329600Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:46.2330030Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:46.2330369Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:46.2330751Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:49:46.2331088Z skip: CUDA is not available. (1.513s)
2022-12-01T10:49:46.2331369Z   test_zero_join_cpu (__main__.TestZeroRedundancyOptimizerDistributed)
2022-12-01T10:49:46.2331784Z Check that the ZeRO join hook allows training with uneven inputs ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12721
2022-12-01T10:49:46.2332187Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12722
2022-12-01T10:49:46.2332722Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:46.2333046Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:46.2333481Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:46.2333836Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:46.2334169Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:49:46.2334622Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:46.2334956Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:46.2335396Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:46.2335728Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:46.2336055Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:49:46.2336424Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:49:46.2336801Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:49:46.2337290Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:49:46.2337818Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:49:46.2338228Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph3xkffc2
2022-12-01T10:49:46.2338625Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgrzinasq
2022-12-01T10:49:46.2339014Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph3xkffc2/_remote_module_non_scriptable.py
2022-12-01T10:49:46.2339427Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgrzinasq/_remote_module_non_scriptable.py
2022-12-01T10:49:46.2339815Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:49:46.2340165Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:49:46.2340855Z /opt/conda/lib/python3.7/tempfile.py:798: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmph3xkffc2'>
2022-12-01T10:49:46.2341202Z   _warnings.warn(warn_message, ResourceWarning)
2022-12-01T10:49:46.2341641Z /opt/conda/lib/python3.7/tempfile.py:798: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmpgrzinasq'>
2022-12-01T10:49:46.2341965Z   _warnings.warn(warn_message, ResourceWarning)
2022-12-01T10:49:46.2342175Z ok (1.511s)
2022-12-01T10:49:46.2342443Z   test_zero_join_gpu (__main__.TestZeroRedundancyOptimizerDistributed)
2022-12-01T10:49:46.2342810Z Check that the ZeRO join hook allows training with uneven inputs ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-12-01T10:49:46.2343234Z   test_zero_model_parallel_parameters_as_bucket_view_False (__main__.TestZeroRedundancyOptimizerDistributed)
2022-12-01T10:49:46.2343858Z Check that ZeRO works with model parallelism where the model's ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12749
2022-12-01T10:49:46.2344259Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12750
2022-12-01T10:49:46.2344745Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:46.2345079Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:46.2345511Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:46.2345846Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:46.2346177Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:49:46.2346638Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:46.2346967Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:46.2347388Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:46.2347737Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:46.2348066Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:49:46.2348357Z skip: Need at least 4 CUDA devices (1.411s)
2022-12-01T10:49:46.2348684Z   test_zero_model_parallel_parameters_as_bucket_view_True (__main__.TestZeroRedundancyOptimizerDistributed)
2022-12-01T10:49:46.2349244Z Check that ZeRO works with model parallelism where the model's ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12771
2022-12-01T10:49:46.2349642Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12772
2022-12-01T10:49:46.2350080Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:46.2350409Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:46.2350843Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:46.2351190Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:46.2351506Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:49:46.2351965Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:46.2352297Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:46.2352715Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:46.2353068Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:46.2353394Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:49:46.2353690Z skip: Need at least 4 CUDA devices (1.412s)
2022-12-01T10:49:46.2353975Z   test_constructor (__main__.TestZeroRedundancyOptimizerSingleRank)
2022-12-01T10:49:46.2354403Z Check the robustness of the ZeroRedundancyOptimizer constructor by ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12793
2022-12-01T10:49:46.2354936Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:46.2355266Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:46.2355685Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:46.2356032Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:46.2356358Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:49:46.2356755Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:49:46.2357289Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-12-01T10:49:46.2357582Z ok (1.409s)
2022-12-01T10:49:46.2357864Z   test_lr_scheduler (__main__.TestZeroRedundancyOptimizerSingleRank)
2022-12-01T10:49:46.2358260Z Check that a normal PyTorch ``lr_scheduler`` is usable with ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12807
2022-12-01T10:49:46.2358765Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:46.2359097Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:46.2359508Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:46.2359856Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:46.2360182Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:49:46.2360547Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:49:46.2361033Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-12-01T10:49:46.2361326Z ok (1.409s)
2022-12-01T10:49:46.2361600Z   test_same_dense_param_type (__main__.TestZeroRedundancyOptimizerSingleRank)
2022-12-01T10:49:46.2362031Z Check that ZeroRedundancyOptimizer raises an exception if the input ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12821
2022-12-01T10:49:46.2362565Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:46.2362897Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:46.2363333Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:46.2363669Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:46.2363998Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:49:46.2364361Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:49:46.2364859Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-12-01T10:49:46.2365138Z ok (1.308s)
2022-12-01T10:49:46.2365397Z   test_state_dict (__main__.TestZeroRedundancyOptimizerSingleRank)
2022-12-01T10:49:46.2365829Z Check that ZeroRedundancyOptimizer exposes the expected state dict ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12835
2022-12-01T10:49:46.2366354Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:46.2366687Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:46.2367118Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:46.2367467Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:46.2367779Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:49:46.2368145Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:49:46.2368644Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-12-01T10:49:46.2368940Z ok (1.309s)
2022-12-01T10:49:46.2369201Z   test_step_with_extra_inner_key (__main__.TestZeroRedundancyOptimizerSingleRank)
2022-12-01T10:49:46.2369679Z Check that ZeroRedundancyOptimizer wrapping an optimizer that adds ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12849
2022-12-01T10:49:46.2370244Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:46.2370563Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:46.2370998Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:46.2371346Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:46.2371672Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:49:46.2372023Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:49:46.2372605Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-12-01T10:49:46.2372900Z ok (1.408s)
2022-12-01T10:49:46.2373154Z   test_step_with_kwargs (__main__.TestZeroRedundancyOptimizerSingleRank)
2022-12-01T10:49:46.2373577Z Check that the ``step(**kwargs)`` interface is properly exposed. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12863
2022-12-01T10:49:46.2374098Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:46.2374430Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:46.2374863Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:46.2375294Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:46.2375620Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:49:46.2376055Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:49:46.2376550Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-12-01T10:49:46.2376844Z ok (1.408s)
2022-12-01T10:49:46.2377115Z   test_step_without_closure (__main__.TestZeroRedundancyOptimizerSingleRank)
2022-12-01T10:49:46.2377520Z Check that the ``step()`` method (without closure) is handled as ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12877
2022-12-01T10:49:46.2378035Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:46.2378365Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:46.2378797Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:46.2379134Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:46.2379461Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:49:46.2379831Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:49:46.2380320Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-12-01T10:49:46.2380830Z ok (1.408s)
2022-12-01T10:49:46.2381089Z   test_zero_grad (__main__.TestZeroRedundancyOptimizerSingleRank)
2022-12-01T10:49:46.2381489Z Check that the ``zero_grad`` method is properly handled. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12891
2022-12-01T10:49:46.2381989Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:46.2382323Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:46.2382759Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:46.2383187Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:46.2383544Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:49:46.2383911Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:49:46.2384414Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-12-01T10:49:46.2384696Z ok (1.408s)
2022-12-01T10:49:46.2384806Z 
2022-12-01T10:49:46.2385011Z ----------------------------------------------------------------------
2022-12-01T10:49:46.2385262Z Ran 42 tests in 32.816s
2022-12-01T10:49:46.2385379Z 
2022-12-01T10:49:46.2385457Z OK (skipped=32)
2022-12-01T10:49:46.2385555Z 
2022-12-01T10:49:46.2385641Z Generating XML reports...
2022-12-01T10:49:46.2386204Z Generated XML report: test-reports/python-unittest/distributed.optim.test_zero_redundancy_optimizer/TEST-TestZeroRedundancyOptimizerDistributed-20221201104913.xml
2022-12-01T10:49:46.2386964Z Generated XML report: test-reports/python-unittest/distributed.optim.test_zero_redundancy_optimizer/TEST-TestZeroRedundancyOptimizerSingleRank-20221201104913.xml
2022-12-01T10:49:46.2387300Z 
2022-12-01T10:49:46.2387648Z ##[endgroup]
2022-12-01T10:49:46.2388154Z FINISHED PRINTING LOG FILE of distributed/optim/test_zero_redundancy_optimizer (/var/lib/jenkins/workspace/test/test-reports/distributed-optim-test_zero_redundancy_optimizer_mur7lrv5)
2022-12-01T10:49:46.2388453Z 
2022-12-01T10:49:46.2388674Z Running distributed/elastic/multiprocessing/api_test ... [2022-12-01 10:49:46.223955]
2022-12-01T10:49:46.2389230Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/elastic/multiprocessing/api_test.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-12-01 10:49:46.224373]
2022-12-01T10:50:04.2400882Z 
2022-12-01T10:50:04.2401312Z Expand the folded group to see the log file of distributed/elastic/multiprocessing/api_test
2022-12-01T10:50:04.2402223Z ##[group]PRINTING LOG FILE of distributed/elastic/multiprocessing/api_test (/var/lib/jenkins/workspace/test/test-reports/distributed-elastic-multiprocessing-api_test_vinj252c)
2022-12-01T10:50:04.2403086Z Test results will be stored in test-reports/python-unittest/distributed.elastic.multiprocessing.api_test
2022-12-01T10:50:04.2403434Z 
2022-12-01T10:50:04.2403540Z Running tests...
2022-12-01T10:50:04.2404049Z ----------------------------------------------------------------------
2022-12-01T10:50:04.2404577Z   test_get_failures (__main__.RunProcResultsTest) ... ok (0.241s)
2022-12-01T10:50:04.2405019Z   test_is_failed (__main__.RunProcResultsTest) ... ok (0.001s)
2022-12-01T10:50:04.2405522Z   test_args_env_len_mismatch (__main__.StartProcessesListTest) ... ok (0.002s)
2022-12-01T10:50:04.2406077Z   test_binary (__main__.StartProcessesListTest) ... hello stderr from 0
2022-12-01T10:50:04.2406519Z hello stdout from 0
2022-12-01T10:50:04.2406843Z hello stderr from 1
2022-12-01T10:50:04.2407155Z hello stdout from 1
2022-12-01T10:50:04.2407404Z ok (0.112s)
2022-12-01T10:50:04.2407649Z   test_binary_exit (__main__.StartProcessesListTest) ... bar stderr from 1
2022-12-01T10:50:04.2407880Z bar stdout from 1
2022-12-01T10:50:04.2408168Z failed (exitcode: 138) local_rank: 0 (pid: 12917) of binary: distributed/elastic/multiprocessing/bin/echo1.py
2022-12-01T10:50:04.2408436Z ok (0.114s)
2022-12-01T10:50:04.2424121Z   test_binary_incorrect_entrypoint (__main__.StartProcessesListTest) ... ok (0.007s)
2022-12-01T10:50:04.2424716Z   test_binary_raises (__main__.StartProcessesListTest) ... Traceback (most recent call last):
2022-12-01T10:50:04.2425261Z   File "distributed/elastic/multiprocessing/bin/echo2.py", line 22, in <module>
2022-12-01T10:50:04.2425705Z     raise RuntimeError(f"raised from {rank}")
2022-12-01T10:50:04.2426035Z RuntimeError: raised from 0
2022-12-01T10:50:04.2426357Z bar from 1
2022-12-01T10:50:04.2427041Z failed (exitcode: 1) local_rank: 0 (pid: 12920) of binary: distributed/elastic/multiprocessing/bin/echo2.py
2022-12-01T10:50:04.2427458Z ok (0.111s)
2022-12-01T10:50:04.2427920Z   test_binary_redirect_and_tee (__main__.StartProcessesListTest) ... world stdout from 1
2022-12-01T10:50:04.2428364Z [trainer1]:world stderr from 1
2022-12-01T10:50:04.2428681Z ok (1.115s)
2022-12-01T10:50:04.2429088Z   test_function (__main__.StartProcessesListTest) ... [trainer0]:hello stdout from 0
2022-12-01T10:50:04.2429942Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:50:04.2430476Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:50:04.2431380Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:50:04.2432030Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:50:04.2432847Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:50:04.2433428Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:50:04.2434232Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:50:04.2434828Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:50:04.2435263Z hello stdout from 1
2022-12-01T10:50:04.2435569Z hello stdout from 0
2022-12-01T10:50:04.2435890Z hello stderr from 1
2022-12-01T10:50:04.2436233Z hello stderr from 0
2022-12-01T10:50:04.2436604Z Closing process 12927 via signal SIGTERM
2022-12-01T10:50:04.2436908Z ok (1.329s)
2022-12-01T10:50:04.2437710Z   test_function_large_ret_val (__main__.StartProcessesListTest) ... /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:50:04.2438371Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:50:04.2439066Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:50:04.2439632Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:50:04.2440334Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:50:04.2440875Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:50:04.2441516Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:50:04.2442028Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:50:04.2442709Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:50:04.2449125Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:50:04.2449881Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:50:04.2450284Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:50:04.2450989Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:50:04.2451575Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:50:04.2452341Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:50:04.2452897Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:50:04.2453315Z Closing process 12947 via signal SIGTERM
2022-12-01T10:50:04.2453702Z Closing process 12948 via signal SIGTERM
2022-12-01T10:50:04.2454100Z Closing process 12949 via signal SIGTERM
2022-12-01T10:50:04.2454447Z ok (1.745s)
2022-12-01T10:50:04.2454837Z   test_function_raise (__main__.StartProcessesListTest)
2022-12-01T10:50:04.2455668Z run 2x copies of echo2, raise an exception on the first ... /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:50:04.2456098Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:50:04.2456533Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:50:04.2456929Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:50:04.2457589Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:50:04.2457921Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:50:04.2458338Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:50:04.2458680Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:50:04.2458989Z failed (exitcode: 1) local_rank: 0 (pid: 12987) of fn: echo2 (start_method: spawn)
2022-12-01T10:50:04.2459253Z Traceback (most recent call last):
2022-12-01T10:50:04.2459640Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/elastic/multiprocessing/api.py", line 455, in _poll
2022-12-01T10:50:04.2459953Z     self._pc.join(-1)
2022-12-01T10:50:04.2460303Z   File "/opt/conda/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 160, in join
2022-12-01T10:50:04.2460812Z     raise ProcessRaisedException(msg, error_index, failed_process.pid)
2022-12-01T10:50:04.2461168Z torch.multiprocessing.spawn.ProcessRaisedException: 
2022-12-01T10:50:04.2461371Z 
2022-12-01T10:50:04.2461541Z -- Process 0 terminated with the following error:
2022-12-01T10:50:04.2461787Z Traceback (most recent call last):
2022-12-01T10:50:04.2462226Z   File "/opt/conda/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 69, in _wrap
2022-12-01T10:50:04.2462490Z     fn(i, *args)
2022-12-01T10:50:04.2462854Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/elastic/multiprocessing/api.py", line 371, in _wrap
2022-12-01T10:50:04.2463150Z     ret = record(fn)(*args_)
2022-12-01T10:50:04.2463557Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 346, in wrapper
2022-12-01T10:50:04.2463844Z     return f(*args, **kwargs)
2022-12-01T10:50:04.2464134Z   File "/var/lib/jenkins/workspace/test/distributed/elastic/multiprocessing/api_test.py", line 138, in echo2
2022-12-01T10:50:04.2464411Z     raise RuntimeError(msg)
2022-12-01T10:50:04.2464592Z RuntimeError: hello
2022-12-01T10:50:04.2464707Z 
2022-12-01T10:50:04.2464775Z ok (1.319s)
2022-12-01T10:50:04.2465019Z   test_function_with_tensor (__main__.StartProcessesListTest) ... ok (0.002s)
2022-12-01T10:50:04.2465331Z   test_invalid_log_dir (__main__.StartProcessesListTest) ... ok (0.001s)
2022-12-01T10:50:04.2465670Z   test_multiprocess_context_close (__main__.StartProcessesListTest) ... Closing process 13007 via signal SIGTERM
2022-12-01T10:50:04.2465937Z ok (0.007s)
2022-12-01T10:50:04.2466399Z   test_multiprocessing_context_poll_raises_exception (__main__.StartProcessesListTest) ... failed (exitcode: -1) local_rank: 0 (pid: 123) of fn: echo0 (start_method: spawn)
2022-12-01T10:50:04.2466732Z Traceback (most recent call last):
2022-12-01T10:50:04.2467165Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/elastic/multiprocessing/api.py", line 455, in _poll
2022-12-01T10:50:04.2467478Z     self._pc.join(-1)
2022-12-01T10:50:04.2467719Z   File "/opt/conda/lib/python3.7/unittest/mock.py", line 1016, in __call__
2022-12-01T10:50:04.2467967Z     return _mock_self._mock_call(*args, **kwargs)
2022-12-01T10:50:04.2468234Z   File "/opt/conda/lib/python3.7/unittest/mock.py", line 1076, in _mock_call
2022-12-01T10:50:04.2468463Z     raise effect
2022-12-01T10:50:04.2468726Z torch.multiprocessing.spawn.ProcessRaisedException: test msg
2022-12-01T10:50:04.2468989Z ok (0.005s)
2022-12-01T10:50:04.2469567Z   test_pcontext_wait (__main__.StartProcessesListTest) ... /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:50:04.2469996Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:50:04.2470429Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:50:04.2470777Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:50:04.2471010Z ok (2.306s)
2022-12-01T10:50:04.2471288Z   test_subprocess_context_close (__main__.StartProcessesListTest) ... Sending process 13018 closing signal SIGTERM
2022-12-01T10:50:04.2471563Z ok (0.009s)
2022-12-01T10:50:04.2471790Z   test_to_map (__main__.StartProcessesListTest) ... ok (0.002s)
2022-12-01T10:50:04.2472071Z   test_validate_full_rank (__main__.StartProcessesListTest) ... ok (0.001s)
2022-12-01T10:50:04.2472615Z   test_void_function (__main__.StartProcessesListTest) ... /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:50:04.2473000Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:50:04.2473431Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:50:04.2473760Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:50:04.2474194Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:50:04.2474520Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:50:04.2474933Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:50:04.2475276Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:50:04.2475498Z hello
2022-12-01T10:50:04.2475646Z world
2022-12-01T10:50:04.2475822Z Closing process 13020 via signal SIGTERM
2022-12-01T10:50:04.2476017Z ok (1.323s)
2022-12-01T10:50:04.2476251Z   test_args_env_len_mismatch (__main__.StartProcessesTest) ... ok (0.002s)
2022-12-01T10:50:04.2476532Z   test_binary_exit (__main__.StartProcessesTest) ... bar stderr from 1
2022-12-01T10:50:04.2476763Z bar stdout from 1
2022-12-01T10:50:04.2477047Z failed (exitcode: 138) local_rank: 0 (pid: 13039) of binary: distributed/elastic/multiprocessing/bin/echo1.py
2022-12-01T10:50:04.2477295Z ok (0.112s)
2022-12-01T10:50:04.2477539Z   test_binary_incorrect_entrypoint (__main__.StartProcessesTest) ... ok (0.007s)
2022-12-01T10:50:04.2477863Z   test_binary_raises (__main__.StartProcessesTest) ... Traceback (most recent call last):
2022-12-01T10:50:04.2478174Z   File "distributed/elastic/multiprocessing/bin/echo2.py", line 22, in <module>
2022-12-01T10:50:04.2478446Z     raise RuntimeError(f"raised from {rank}")
2022-12-01T10:50:04.2478665Z RuntimeError: raised from 0
2022-12-01T10:50:04.2478851Z bar from 1
2022-12-01T10:50:04.2479111Z failed (exitcode: 1) local_rank: 0 (pid: 13042) of binary: distributed/elastic/multiprocessing/bin/echo2.py
2022-12-01T10:50:04.2479373Z ok (0.112s)
2022-12-01T10:50:04.2479841Z   test_function_large_ret_val (__main__.StartProcessesTest) ... /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:50:04.2480215Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:50:04.2480647Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:50:04.2480995Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:50:04.2481422Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:50:04.2481733Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:50:04.2482202Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:50:04.2482584Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:50:04.2483001Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:50:04.2483333Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:50:04.2483761Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:50:04.2484109Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:50:04.2484528Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:50:04.2484854Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:50:04.2485279Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:50:04.2485621Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:50:04.2485863Z Closing process 13044 via signal SIGTERM
2022-12-01T10:50:04.2486090Z Closing process 13045 via signal SIGTERM
2022-12-01T10:50:04.2486310Z Closing process 13047 via signal SIGTERM
2022-12-01T10:50:04.2486487Z ok (1.730s)
2022-12-01T10:50:04.2486702Z   test_function_raise (__main__.StartProcessesTest)
2022-12-01T10:50:04.2487214Z run 2x copies of echo2, raise an exception on the first ... /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:50:04.2487580Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:50:04.2488010Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:50:04.2488358Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:50:04.2488789Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:50:04.2489108Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:50:04.2489531Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:50:04.2489875Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:50:04.2490164Z failed (exitcode: 1) local_rank: 0 (pid: 13084) of fn: echo2 (start_method: spawn)
2022-12-01T10:50:04.2490422Z Traceback (most recent call last):
2022-12-01T10:50:04.2490821Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/elastic/multiprocessing/api.py", line 455, in _poll
2022-12-01T10:50:04.2491136Z     self._pc.join(-1)
2022-12-01T10:50:04.2491474Z   File "/opt/conda/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 160, in join
2022-12-01T10:50:04.2491817Z     raise ProcessRaisedException(msg, error_index, failed_process.pid)
2022-12-01T10:50:04.2492248Z torch.multiprocessing.spawn.ProcessRaisedException: 
2022-12-01T10:50:04.2492451Z 
2022-12-01T10:50:04.2492604Z -- Process 0 terminated with the following error:
2022-12-01T10:50:04.2492844Z Traceback (most recent call last):
2022-12-01T10:50:04.2493217Z   File "/opt/conda/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 69, in _wrap
2022-12-01T10:50:04.2493483Z     fn(i, *args)
2022-12-01T10:50:04.2493848Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/elastic/multiprocessing/api.py", line 371, in _wrap
2022-12-01T10:50:04.2494139Z     ret = record(fn)(*args_)
2022-12-01T10:50:04.2494547Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 346, in wrapper
2022-12-01T10:50:04.2494835Z     return f(*args, **kwargs)
2022-12-01T10:50:04.2495171Z   File "/var/lib/jenkins/workspace/test/distributed/elastic/multiprocessing/api_test.py", line 138, in echo2
2022-12-01T10:50:04.2495490Z     raise RuntimeError(msg)
2022-12-01T10:50:04.2495673Z RuntimeError: hello
2022-12-01T10:50:04.2495790Z 
2022-12-01T10:50:04.2495860Z ok (1.320s)
2022-12-01T10:50:04.2496102Z   test_function_with_tensor (__main__.StartProcessesTest) ... ok (0.002s)
2022-12-01T10:50:04.2496403Z   test_invalid_log_dir (__main__.StartProcessesTest) ... ok (0.002s)
2022-12-01T10:50:04.2496726Z   test_multiprocess_context_close (__main__.StartProcessesTest) ... Closing process 13104 via signal SIGTERM
2022-12-01T10:50:04.2496991Z ok (0.007s)
2022-12-01T10:50:04.2497451Z   test_multiprocessing_context_poll_raises_exception (__main__.StartProcessesTest) ... failed (exitcode: -1) local_rank: 0 (pid: 123) of fn: echo0 (start_method: spawn)
2022-12-01T10:50:04.2497783Z Traceback (most recent call last):
2022-12-01T10:50:04.2498186Z   File "/opt/conda/lib/python3.7/site-packages/torch/distributed/elastic/multiprocessing/api.py", line 455, in _poll
2022-12-01T10:50:04.2498504Z     self._pc.join(-1)
2022-12-01T10:50:04.2498733Z   File "/opt/conda/lib/python3.7/unittest/mock.py", line 1016, in __call__
2022-12-01T10:50:04.2499000Z     return _mock_self._mock_call(*args, **kwargs)
2022-12-01T10:50:04.2499265Z   File "/opt/conda/lib/python3.7/unittest/mock.py", line 1076, in _mock_call
2022-12-01T10:50:04.2499498Z     raise effect
2022-12-01T10:50:04.2499761Z torch.multiprocessing.spawn.ProcessRaisedException: test msg
2022-12-01T10:50:04.2500025Z ok (0.004s)
2022-12-01T10:50:04.2500703Z   test_pcontext_wait (__main__.StartProcessesTest) ... /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:50:04.2501207Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:50:04.2502022Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:50:04.2502560Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:50:04.2502910Z ok (2.304s)
2022-12-01T10:50:04.2503307Z   test_subprocess_context_close (__main__.StartProcessesTest) ... Sending process 13115 closing signal SIGTERM
2022-12-01T10:50:04.2503797Z ok (0.008s)
2022-12-01T10:50:04.2504130Z   test_to_map (__main__.StartProcessesTest) ... ok (0.002s)
2022-12-01T10:50:04.2504405Z   test_validate_full_rank (__main__.StartProcessesTest) ... ok (0.001s)
2022-12-01T10:50:04.2504942Z   test_void_function (__main__.StartProcessesTest) ... /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:50:04.2505326Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:50:04.2505762Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:50:04.2506097Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:50:04.2506534Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:50:04.2506865Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:50:04.2507275Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:50:04.2507620Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:50:04.2507844Z world
2022-12-01T10:50:04.2507997Z hello
2022-12-01T10:50:04.2508168Z Closing process 13116 via signal SIGTERM
2022-12-01T10:50:04.2508363Z ok (1.325s)
2022-12-01T10:50:04.2508576Z   test_from_str_bad_input (__main__.StdTest) ... ok (0.002s)
2022-12-01T10:50:04.2508810Z   test_from_value (__main__.StdTest) ... ok (0.001s)
2022-12-01T10:50:04.2509051Z   test_from_value_map (__main__.StdTest) ... ok (0.001s)
2022-12-01T10:50:04.2509192Z 
2022-12-01T10:50:04.2509393Z ----------------------------------------------------------------------
2022-12-01T10:50:04.2509721Z Ran 38 tests in 16.701s
2022-12-01T10:50:04.2509840Z 
2022-12-01T10:50:04.2509957Z OK
2022-12-01T10:50:04.2510051Z 
2022-12-01T10:50:04.2510141Z Generating XML reports...
2022-12-01T10:50:04.2510626Z Generated XML report: test-reports/python-unittest/distributed.elastic.multiprocessing.api_test/TEST-RunProcResultsTest-20221201104947.xml
2022-12-01T10:50:04.2511305Z Generated XML report: test-reports/python-unittest/distributed.elastic.multiprocessing.api_test/TEST-StartProcessesListTest-20221201104947.xml
2022-12-01T10:50:04.2511969Z Generated XML report: test-reports/python-unittest/distributed.elastic.multiprocessing.api_test/TEST-StartProcessesTest-20221201104947.xml
2022-12-01T10:50:04.2512585Z Generated XML report: test-reports/python-unittest/distributed.elastic.multiprocessing.api_test/TEST-StdTest-20221201104947.xml
2022-12-01T10:50:04.2512848Z 
2022-12-01T10:50:04.2513232Z ##[endgroup]
2022-12-01T10:50:04.2513753Z FINISHED PRINTING LOG FILE of distributed/elastic/multiprocessing/api_test (/var/lib/jenkins/workspace/test/test-reports/distributed-elastic-multiprocessing-api_test_vinj252c)
2022-12-01T10:50:04.2514064Z 
2022-12-01T10:50:04.2514253Z Running distributed/test_pg_wrapper ... [2022-12-01 10:50:04.240605]
2022-12-01T10:50:04.2514781Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_pg_wrapper.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-12-01 10:50:04.240934]
2022-12-01T10:50:46.5358829Z 
2022-12-01T10:50:46.5359521Z Expand the folded group to see the log file of distributed/test_pg_wrapper
2022-12-01T10:50:46.5361092Z ##[group]PRINTING LOG FILE of distributed/test_pg_wrapper (/var/lib/jenkins/workspace/test/test-reports/distributed-test_pg_wrapper_gqx32mdz)
2022-12-01T10:50:46.5381423Z <unittest.suite.TestSuite tests=[]>
2022-12-01T10:50:46.5381900Z <unittest.suite.TestSuite tests=[]>
2022-12-01T10:50:46.5383262Z <unittest.suite.TestSuite tests=[<__main__.ProcessGroupGlooWrapperTest testMethod=test_collective_hang>, <__main__.ProcessGroupGlooWrapperTest testMethod=test_collective_shape_mismatch>, <__main__.ProcessGroupGlooWrapperTest testMethod=test_collective_shape_mismatch_cuda>, <__main__.ProcessGroupGlooWrapperTest testMethod=test_collective_shape_mismatch_cuda_debug_mode>, <__main__.ProcessGroupGlooWrapperTest testMethod=test_collective_shape_mismatch_debug_mode>, <__main__.ProcessGroupGlooWrapperTest testMethod=test_collectives_op_mismatch>, <__main__.ProcessGroupGlooWrapperTest testMethod=test_collectives_op_mismatch_cuda>, <__main__.ProcessGroupGlooWrapperTest testMethod=test_collectives_op_mismatch_cuda_debug_mode>, <__main__.ProcessGroupGlooWrapperTest testMethod=test_collectives_op_mismatch_debug_mode>]>
2022-12-01T10:50:46.5384456Z test_collective_hang (__main__.ProcessGroupGlooWrapperTest)
2022-12-01T10:50:46.5384785Z test_collective_shape_mismatch (__main__.ProcessGroupGlooWrapperTest)
2022-12-01T10:50:46.5385124Z test_collective_shape_mismatch_cuda (__main__.ProcessGroupGlooWrapperTest)
2022-12-01T10:50:46.5391960Z test_collective_shape_mismatch_cuda_debug_mode (__main__.ProcessGroupGlooWrapperTest)
2022-12-01T10:50:46.5392373Z test_collective_shape_mismatch_debug_mode (__main__.ProcessGroupGlooWrapperTest)
2022-12-01T10:50:46.5392718Z test_collectives_op_mismatch (__main__.ProcessGroupGlooWrapperTest)
2022-12-01T10:50:46.5393050Z test_collectives_op_mismatch_cuda (__main__.ProcessGroupGlooWrapperTest)
2022-12-01T10:50:46.5393384Z test_collectives_op_mismatch_cuda_debug_mode (__main__.ProcessGroupGlooWrapperTest)
2022-12-01T10:50:46.5393744Z test_collectives_op_mismatch_debug_mode (__main__.ProcessGroupGlooWrapperTest)
2022-12-01T10:50:46.5394792Z <unittest.suite.TestSuite tests=[<__main__.ProcessGroupNCCLWrapperTest testMethod=test_collective_hang>, <__main__.ProcessGroupNCCLWrapperTest testMethod=test_collective_shape_mismatch>, <__main__.ProcessGroupNCCLWrapperTest testMethod=test_collective_shape_mismatch_debug_mode>, <__main__.ProcessGroupNCCLWrapperTest testMethod=test_collectives_op_mismatch>, <__main__.ProcessGroupNCCLWrapperTest testMethod=test_collectives_op_mismatch_debug_mode>]>
2022-12-01T10:50:46.5396397Z test_collective_hang (__main__.ProcessGroupNCCLWrapperTest)
2022-12-01T10:50:46.5397047Z test_collective_shape_mismatch (__main__.ProcessGroupNCCLWrapperTest)
2022-12-01T10:50:46.5397811Z test_collective_shape_mismatch_debug_mode (__main__.ProcessGroupNCCLWrapperTest)
2022-12-01T10:50:46.5398529Z test_collectives_op_mismatch (__main__.ProcessGroupNCCLWrapperTest)
2022-12-01T10:50:46.5399283Z test_collectives_op_mismatch_debug_mode (__main__.ProcessGroupNCCLWrapperTest)
2022-12-01T10:50:46.5400364Z Test results will be stored in test-reports/python-unittest/distributed.test_pg_wrapper
2022-12-01T10:50:46.5401393Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:50:46.5402125Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:50:46.5403092Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:50:46.5403858Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:50:46.5404195Z 
2022-12-01T10:50:46.5404352Z Running tests...
2022-12-01T10:50:46.5405010Z ----------------------------------------------------------------------
2022-12-01T10:50:46.5405898Z   test_collective_hang (__main__.ProcessGroupGlooWrapperTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13157
2022-12-01T10:50:46.5406774Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13158
2022-12-01T10:50:46.5407507Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13159
2022-12-01T10:50:46.5408220Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 13160
2022-12-01T10:50:46.5409239Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:50:46.5409938Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:50:46.5410886Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:50:46.5411652Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:50:46.5412397Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:50:46.5416056Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:50:46.5416566Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:50:46.5417394Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:50:46.5417844Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:50:46.5418253Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:50:46.5418722Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:50:46.5419067Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:50:46.5419794Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:50:46.5420475Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:50:46.5421006Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:50:46.5421740Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:50:46.5422075Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:50:46.5422493Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:50:46.5422953Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:50:46.5423333Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:50:46.5423704Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:50:46.5424065Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:50:46.5424434Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:50:46.5424811Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-12-01T10:50:46.5425298Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:50:46.5425824Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:50:46.5426349Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:50:46.5426872Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:50:46.5427249Z [E ProcessGroupGloo.cpp:2803] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 2000 ms
2022-12-01T10:50:46.5427600Z [E ProcessGroupGloo.cpp:137] [Rank 0]: Ranks 1 failed to pass monitoredBarrier in 2000 ms
2022-12-01T10:50:46.5428044Z [E ProcessGroupGloo.cpp:137] Rank 2 successfully reached monitoredBarrier, but received errors while waiting for send/recv from rank 0. Please check rank 0 logs for faulty rank.
2022-12-01T10:50:46.5428566Z [E ProcessGroupGloo.cpp:137] Rank 3 successfully reached monitoredBarrier, but received errors while waiting for send/recv from rank 0. Please check rank 0 logs for faulty rank.
2022-12-01T10:50:46.5428883Z ok (2.172s)
2022-12-01T10:50:46.5428990Z 
2022-12-01T10:50:46.5429195Z ----------------------------------------------------------------------
2022-12-01T10:50:46.5429442Z Ran 1 test in 2.172s
2022-12-01T10:50:46.5429559Z 
2022-12-01T10:50:46.5429622Z OK
2022-12-01T10:50:46.5429701Z 
2022-12-01T10:50:46.5429786Z Generating XML reports...
2022-12-01T10:50:46.5430267Z Generated XML report: test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20221201105006.xml
2022-12-01T10:50:46.5430801Z Test results will be stored in test-reports/python-unittest/distributed.test_pg_wrapper
2022-12-01T10:50:46.5431269Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:50:46.5431591Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:50:46.5432024Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:50:46.5432378Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:50:46.5432550Z 
2022-12-01T10:50:46.5432629Z Running tests...
2022-12-01T10:50:46.5432954Z ----------------------------------------------------------------------
2022-12-01T10:50:46.5433374Z   test_collective_shape_mismatch (__main__.ProcessGroupGlooWrapperTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13248
2022-12-01T10:50:46.5433795Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13249
2022-12-01T10:50:46.5434112Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13250
2022-12-01T10:50:46.5434440Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 13251
2022-12-01T10:50:46.5434891Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:50:46.5435230Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:50:46.5435690Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:50:46.5436072Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:50:46.5436400Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:50:46.5436849Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:50:46.5437186Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:50:46.5437617Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:50:46.5437964Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:50:46.5438276Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:50:46.5438744Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:50:46.5439077Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:50:46.5439496Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:50:46.5439846Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:50:46.5440176Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:50:46.5440641Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:50:46.5440959Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:50:46.5441391Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:50:46.5441739Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:50:46.5442071Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:50:46.5442426Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:50:46.5442799Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:50:46.5443163Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:50:46.5443645Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:50:46.5444045Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-12-01T10:50:46.5444537Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:50:46.5445055Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:50:46.5445560Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:50:46.5445850Z ok (2.063s)
2022-12-01T10:50:46.5445955Z 
2022-12-01T10:50:46.5446153Z ----------------------------------------------------------------------
2022-12-01T10:50:46.5446397Z Ran 1 test in 2.064s
2022-12-01T10:50:46.5446498Z 
2022-12-01T10:50:46.5446560Z OK
2022-12-01T10:50:46.5446653Z 
2022-12-01T10:50:46.5446741Z Generating XML reports...
2022-12-01T10:50:46.5447218Z Generated XML report: test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20221201105010.xml
2022-12-01T10:50:46.5447716Z Test results will be stored in test-reports/python-unittest/distributed.test_pg_wrapper
2022-12-01T10:50:46.5448182Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:50:46.5448576Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:50:46.5449007Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:50:46.5449374Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:50:46.5449538Z 
2022-12-01T10:50:46.5453018Z Running tests...
2022-12-01T10:50:46.5453598Z ----------------------------------------------------------------------
2022-12-01T10:50:46.5454319Z   test_collective_shape_mismatch_cuda (__main__.ProcessGroupGlooWrapperTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13339
2022-12-01T10:50:46.5455138Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13340
2022-12-01T10:50:46.5455758Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13341
2022-12-01T10:50:46.5456265Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 13342
2022-12-01T10:50:46.5456754Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:50:46.5457097Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:50:46.5457517Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:50:46.5457872Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:50:46.5458205Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:50:46.5458669Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:50:46.5458985Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:50:46.5459417Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:50:46.5459772Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:50:46.5460101Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:50:46.5460750Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:50:46.5461103Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:50:46.5461817Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:50:46.5462361Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:50:46.5462884Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:50:46.5463637Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:50:46.5464434Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:50:46.5465188Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:50:46.5465838Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:50:46.5466441Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:50:46.5466933Z skip: Need at least 4 CUDA devices (1.964s)
2022-12-01T10:50:46.5467072Z 
2022-12-01T10:50:46.5467277Z ----------------------------------------------------------------------
2022-12-01T10:50:46.5467522Z Ran 1 test in 1.964s
2022-12-01T10:50:46.5467637Z 
2022-12-01T10:50:46.5467711Z OK (skipped=1)
2022-12-01T10:50:46.5467806Z 
2022-12-01T10:50:46.5467894Z Generating XML reports...
2022-12-01T10:50:46.5468372Z Generated XML report: test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20221201105014.xml
2022-12-01T10:50:46.5468882Z Test results will be stored in test-reports/python-unittest/distributed.test_pg_wrapper
2022-12-01T10:50:46.5469473Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:50:46.5469847Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:50:46.5470284Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:50:46.5470634Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:50:46.5470801Z 
2022-12-01T10:50:46.5470861Z Running tests...
2022-12-01T10:50:46.5471169Z ----------------------------------------------------------------------
2022-12-01T10:50:46.5471603Z   test_collective_shape_mismatch_cuda_debug_mode (__main__.ProcessGroupGlooWrapperTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13394
2022-12-01T10:50:46.5472041Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13395
2022-12-01T10:50:46.5472509Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13396
2022-12-01T10:50:46.5472835Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 13397
2022-12-01T10:50:46.5473297Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:50:46.5473631Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:50:46.5474051Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:50:46.5474400Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:50:46.5474726Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:50:46.5475172Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:50:46.5475499Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:50:46.5475933Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:50:46.5476282Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:50:46.5476594Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:50:46.5477055Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:50:46.5477383Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:50:46.5477799Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:50:46.5478147Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:50:46.5478472Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:50:46.5478937Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:50:46.5479253Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:50:46.5479682Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:50:46.5480028Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:50:46.5480338Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:50:46.5480630Z skip: Need at least 4 CUDA devices (1.963s)
2022-12-01T10:50:46.5480768Z 
2022-12-01T10:50:46.5480966Z ----------------------------------------------------------------------
2022-12-01T10:50:46.5481208Z Ran 1 test in 1.963s
2022-12-01T10:50:46.5481309Z 
2022-12-01T10:50:46.5481384Z OK (skipped=1)
2022-12-01T10:50:46.5481493Z 
2022-12-01T10:50:46.5481581Z Generating XML reports...
2022-12-01T10:50:46.5482103Z Generated XML report: test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20221201105017.xml
2022-12-01T10:50:46.5482638Z Test results will be stored in test-reports/python-unittest/distributed.test_pg_wrapper
2022-12-01T10:50:46.5483107Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:50:46.5483439Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:50:46.5483870Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:50:46.5484201Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:50:46.5484371Z 
2022-12-01T10:50:46.5484444Z Running tests...
2022-12-01T10:50:46.5484754Z ----------------------------------------------------------------------
2022-12-01T10:50:46.5485191Z   test_collective_shape_mismatch_debug_mode (__main__.ProcessGroupGlooWrapperTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13449
2022-12-01T10:50:46.5485611Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13450
2022-12-01T10:50:46.5485949Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13451
2022-12-01T10:50:46.5486280Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 13452
2022-12-01T10:50:46.5486723Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:50:46.5487053Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:50:46.5487484Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:50:46.5487835Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:50:46.5488147Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:50:46.5488610Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:50:46.5488946Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:50:46.5489368Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:50:46.5489715Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:50:46.5490044Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:50:46.5490507Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:50:46.5490823Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:50:46.5491252Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:50:46.5491599Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:50:46.5491967Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:50:46.5492506Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:50:46.5492837Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:50:46.5493267Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:50:46.5493602Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:50:46.5493930Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:50:46.5494296Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:50:46.5494673Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:50:46.5495202Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:50:46.5495630Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-12-01T10:50:46.5495997Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:50:46.5496492Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:50:46.5496995Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:50:46.5497513Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:50:46.5497914Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T10:50:46.5498272Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-12-01T10:50:46.5498639Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T10:50:46.5499002Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 3
2022-12-01T10:50:46.5499493Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes.
2022-12-01T10:50:46.5499997Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes.
2022-12-01T10:50:46.5500756Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes.
2022-12-01T10:50:46.5501284Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes.
2022-12-01T10:50:46.5501581Z ok (2.269s)
2022-12-01T10:50:46.5501689Z 
2022-12-01T10:50:46.5501882Z ----------------------------------------------------------------------
2022-12-01T10:50:46.5502131Z Ran 1 test in 2.269s
2022-12-01T10:50:46.5502246Z 
2022-12-01T10:50:46.5502309Z OK
2022-12-01T10:50:46.5502397Z 
2022-12-01T10:50:46.5502485Z Generating XML reports...
2022-12-01T10:50:46.5502967Z Generated XML report: test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20221201105021.xml
2022-12-01T10:50:46.5503482Z Test results will be stored in test-reports/python-unittest/distributed.test_pg_wrapper
2022-12-01T10:50:46.5503936Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:50:46.5504271Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:50:46.5504706Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:50:46.5505048Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:50:46.5505219Z 
2022-12-01T10:50:46.5505294Z Running tests...
2022-12-01T10:50:46.5505600Z ----------------------------------------------------------------------
2022-12-01T10:50:46.5506018Z   test_collectives_op_mismatch (__main__.ProcessGroupGlooWrapperTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13552
2022-12-01T10:50:46.5506421Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13553
2022-12-01T10:50:46.5506754Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13554
2022-12-01T10:50:46.5507079Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 13555
2022-12-01T10:50:46.5507531Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:50:46.5507849Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:50:46.5508380Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:50:46.5508775Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:50:46.5509086Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:50:46.5509550Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:50:46.5509880Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:50:46.5510311Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:50:46.5510642Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:50:46.5510967Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:50:46.5511430Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:50:46.5511747Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:50:46.5512173Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:50:46.5512520Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:50:46.5512845Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:50:46.5513290Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:50:46.5513621Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:50:46.5514048Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:50:46.5514393Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:50:46.5514706Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:50:46.5515071Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:50:46.5515443Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:50:46.5515792Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-12-01T10:50:46.5516153Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:50:46.5516646Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:50:46.5517164Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:50:46.5517668Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:50:46.5518185Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:50:46.5518481Z ok (2.265s)
2022-12-01T10:50:46.5518586Z 
2022-12-01T10:50:46.5518786Z ----------------------------------------------------------------------
2022-12-01T10:50:46.5519015Z Ran 1 test in 2.266s
2022-12-01T10:50:46.5519129Z 
2022-12-01T10:50:46.5519191Z OK
2022-12-01T10:50:46.5519284Z 
2022-12-01T10:50:46.5519370Z Generating XML reports...
2022-12-01T10:50:46.5519832Z Generated XML report: test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20221201105025.xml
2022-12-01T10:50:46.5520348Z Test results will be stored in test-reports/python-unittest/distributed.test_pg_wrapper
2022-12-01T10:50:46.5520815Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:50:46.5521185Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:50:46.5521638Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:50:46.5521984Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:50:46.5522147Z 
2022-12-01T10:50:46.5522220Z Running tests...
2022-12-01T10:50:46.5522511Z ----------------------------------------------------------------------
2022-12-01T10:50:46.5522931Z   test_collectives_op_mismatch_cuda (__main__.ProcessGroupGlooWrapperTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13643
2022-12-01T10:50:46.5523354Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13644
2022-12-01T10:50:46.5523685Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13645
2022-12-01T10:50:46.5524030Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 13646
2022-12-01T10:50:46.5524484Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:50:46.5524820Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:50:46.5525243Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:50:46.5525591Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:50:46.5525920Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:50:46.5526381Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:50:46.5526696Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:50:46.5527125Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:50:46.5527473Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:50:46.5527799Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:50:46.5528254Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:50:46.5528583Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:50:46.5529009Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:50:46.5529339Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:50:46.5529663Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:50:46.5530119Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:50:46.5530446Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:50:46.5530859Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:50:46.5531207Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:50:46.5531531Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:50:46.5531826Z skip: Need at least 4 CUDA devices (1.966s)
2022-12-01T10:50:46.5531964Z 
2022-12-01T10:50:46.5532240Z ----------------------------------------------------------------------
2022-12-01T10:50:46.5532485Z Ran 1 test in 1.966s
2022-12-01T10:50:46.5532600Z 
2022-12-01T10:50:46.5532673Z OK (skipped=1)
2022-12-01T10:50:46.5532768Z 
2022-12-01T10:50:46.5532855Z Generating XML reports...
2022-12-01T10:50:46.5533335Z Generated XML report: test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20221201105029.xml
2022-12-01T10:50:46.5533897Z Test results will be stored in test-reports/python-unittest/distributed.test_pg_wrapper
2022-12-01T10:50:46.5534350Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:50:46.5534722Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:50:46.5535155Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:50:46.5535504Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:50:46.5535669Z 
2022-12-01T10:50:46.5535728Z Running tests...
2022-12-01T10:50:46.5536040Z ----------------------------------------------------------------------
2022-12-01T10:50:46.5536472Z   test_collectives_op_mismatch_cuda_debug_mode (__main__.ProcessGroupGlooWrapperTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13698
2022-12-01T10:50:46.5536901Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13699
2022-12-01T10:50:46.5537226Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13700
2022-12-01T10:50:46.5537551Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 13701
2022-12-01T10:50:46.5538003Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:50:46.5538320Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:50:46.5538756Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:50:46.5539103Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:50:46.5539431Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:50:46.5539880Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:50:46.5540209Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:50:46.5540786Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:50:46.5541123Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:50:46.5541449Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:50:46.5541911Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:50:46.5542242Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:50:46.5542660Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:50:46.5543008Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:50:46.5543332Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:50:46.5543794Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:50:46.5544109Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:50:46.5544538Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:50:46.5544885Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:50:46.5545195Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:50:46.5545487Z skip: Need at least 4 CUDA devices (1.869s)
2022-12-01T10:50:46.5545625Z 
2022-12-01T10:50:46.5545827Z ----------------------------------------------------------------------
2022-12-01T10:50:46.5546069Z Ran 1 test in 1.869s
2022-12-01T10:50:46.5546169Z 
2022-12-01T10:50:46.5546242Z OK (skipped=1)
2022-12-01T10:50:46.5546351Z 
2022-12-01T10:50:46.5546435Z Generating XML reports...
2022-12-01T10:50:46.5547019Z Generated XML report: test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20221201105032.xml
2022-12-01T10:50:46.5547562Z Test results will be stored in test-reports/python-unittest/distributed.test_pg_wrapper
2022-12-01T10:50:46.5548026Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:50:46.5548356Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:50:46.5548782Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:50:46.5549114Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:50:46.5549282Z 
2022-12-01T10:50:46.5549358Z Running tests...
2022-12-01T10:50:46.5549668Z ----------------------------------------------------------------------
2022-12-01T10:50:46.5550078Z   test_collectives_op_mismatch_debug_mode (__main__.ProcessGroupGlooWrapperTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13753
2022-12-01T10:50:46.5550511Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13754
2022-12-01T10:50:46.5550842Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13755
2022-12-01T10:50:46.5551175Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 13756
2022-12-01T10:50:46.5551617Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:50:46.5551949Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:50:46.5552377Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:50:46.5552724Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:50:46.5553034Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:50:46.5553498Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:50:46.5553828Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:50:46.5554242Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:50:46.5554587Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:50:46.5554912Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:50:46.5555370Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:50:46.5555685Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:50:46.5556114Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:50:46.5556463Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:50:46.5556775Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:50:46.5557241Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:50:46.5557571Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:50:46.5558001Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:50:46.5558334Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:50:46.5558658Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:50:46.5559024Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:50:46.5559395Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:50:46.5559795Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-12-01T10:50:46.5560203Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:50:46.5560699Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:50:46.5561204Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:50:46.5561726Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:50:46.5562247Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:50:46.5562649Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T10:50:46.5563008Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T10:50:46.5563379Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 3
2022-12-01T10:50:46.5563741Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2022-12-01T10:50:46.5564236Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes.
2022-12-01T10:50:46.5564738Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes.
2022-12-01T10:50:46.5565254Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes.
2022-12-01T10:50:46.5565769Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes.
2022-12-01T10:50:46.5566062Z ok (2.366s)
2022-12-01T10:50:46.5566155Z 
2022-12-01T10:50:46.5566356Z ----------------------------------------------------------------------
2022-12-01T10:50:46.5566607Z Ran 1 test in 2.366s
2022-12-01T10:50:46.5566716Z 
2022-12-01T10:50:46.5566779Z OK
2022-12-01T10:50:46.5566871Z 
2022-12-01T10:50:46.5566955Z Generating XML reports...
2022-12-01T10:50:46.5567431Z Generated XML report: test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20221201105036.xml
2022-12-01T10:50:46.5567970Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:50:46.5568302Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:50:46.5568729Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:50:46.5569074Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:50:46.5569229Z 
2022-12-01T10:50:46.5569303Z Running tests...
2022-12-01T10:50:46.5569602Z ----------------------------------------------------------------------
2022-12-01T10:50:46.5570130Z   test_collective_hang (__main__.ProcessGroupNCCLWrapperTest) ... Test results will be stored in test-reports/python-unittest/distributed.test_pg_wrapper
2022-12-01T10:50:46.5570499Z skip: c10d was not compiled with the NCCL backend (0.000s)
2022-12-01T10:50:46.5570651Z 
2022-12-01T10:50:46.5570846Z ----------------------------------------------------------------------
2022-12-01T10:50:46.5571083Z Ran 1 test in 0.000s
2022-12-01T10:50:46.5571197Z 
2022-12-01T10:50:46.5571267Z OK (skipped=1)
2022-12-01T10:50:46.5571361Z 
2022-12-01T10:50:46.5571445Z Generating XML reports...
2022-12-01T10:50:46.5586582Z Generated XML report: test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupNCCLWrapperTest-20221201105040.xml
2022-12-01T10:50:46.5587313Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:50:46.5587805Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:50:46.5588313Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:50:46.5588652Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:50:46.5588825Z 
2022-12-01T10:50:46.5588901Z Running tests...
2022-12-01T10:50:46.5589211Z ----------------------------------------------------------------------
2022-12-01T10:50:46.5589753Z   test_collective_shape_mismatch (__main__.ProcessGroupNCCLWrapperTest) ... Test results will be stored in test-reports/python-unittest/distributed.test_pg_wrapper
2022-12-01T10:50:46.5590139Z skip: c10d was not compiled with the NCCL backend (0.000s)
2022-12-01T10:50:46.5590289Z 
2022-12-01T10:50:46.5590484Z ----------------------------------------------------------------------
2022-12-01T10:50:46.5590724Z Ran 1 test in 0.000s
2022-12-01T10:50:46.5590836Z 
2022-12-01T10:50:46.5590910Z OK (skipped=1)
2022-12-01T10:50:46.5591006Z 
2022-12-01T10:50:46.5591092Z Generating XML reports...
2022-12-01T10:50:46.5591563Z Generated XML report: test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupNCCLWrapperTest-20221201105041.xml
2022-12-01T10:50:46.5592111Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:50:46.5592430Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:50:46.5592858Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:50:46.5593204Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:50:46.5593368Z 
2022-12-01T10:50:46.5593440Z Running tests...
2022-12-01T10:50:46.5593726Z ----------------------------------------------------------------------
2022-12-01T10:50:46.5594280Z   test_collective_shape_mismatch_debug_mode (__main__.ProcessGroupNCCLWrapperTest) ... Test results will be stored in test-reports/python-unittest/distributed.test_pg_wrapper
2022-12-01T10:50:46.5594684Z skip: c10d was not compiled with the NCCL backend (0.000s)
2022-12-01T10:50:46.5594834Z 
2022-12-01T10:50:46.5595029Z ----------------------------------------------------------------------
2022-12-01T10:50:46.5595256Z Ran 1 test in 0.000s
2022-12-01T10:50:46.5595368Z 
2022-12-01T10:50:46.5595439Z OK (skipped=1)
2022-12-01T10:50:46.5595546Z 
2022-12-01T10:50:46.5595631Z Generating XML reports...
2022-12-01T10:50:46.5596087Z Generated XML report: test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupNCCLWrapperTest-20221201105043.xml
2022-12-01T10:50:46.5596639Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:50:46.5596968Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:50:46.5597395Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:50:46.5597730Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:50:46.5597893Z 
2022-12-01T10:50:46.5597965Z Running tests...
2022-12-01T10:50:46.5598265Z ----------------------------------------------------------------------
2022-12-01T10:50:46.5598782Z   test_collectives_op_mismatch (__main__.ProcessGroupNCCLWrapperTest) ... Test results will be stored in test-reports/python-unittest/distributed.test_pg_wrapper
2022-12-01T10:50:46.5599170Z skip: c10d was not compiled with the NCCL backend (0.000s)
2022-12-01T10:50:46.5599319Z 
2022-12-01T10:50:46.5599513Z ----------------------------------------------------------------------
2022-12-01T10:50:46.5599751Z Ran 1 test in 0.000s
2022-12-01T10:50:46.5599850Z 
2022-12-01T10:50:46.5599920Z OK (skipped=1)
2022-12-01T10:50:46.5600026Z 
2022-12-01T10:50:46.5600110Z Generating XML reports...
2022-12-01T10:50:46.5600638Z Generated XML report: test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupNCCLWrapperTest-20221201105044.xml
2022-12-01T10:50:46.5601205Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:50:46.5601535Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:50:46.5601962Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:50:46.5602311Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:50:46.5602476Z 
2022-12-01T10:50:46.5602535Z Running tests...
2022-12-01T10:50:46.5602837Z ----------------------------------------------------------------------
2022-12-01T10:50:46.5603394Z   test_collectives_op_mismatch_debug_mode (__main__.ProcessGroupNCCLWrapperTest) ... Test results will be stored in test-reports/python-unittest/distributed.test_pg_wrapper
2022-12-01T10:50:46.5603800Z skip: c10d was not compiled with the NCCL backend (0.000s)
2022-12-01T10:50:46.5603938Z 
2022-12-01T10:50:46.5604134Z ----------------------------------------------------------------------
2022-12-01T10:50:46.5604376Z Ran 1 test in 0.000s
2022-12-01T10:50:46.5604492Z 
2022-12-01T10:50:46.5604563Z OK (skipped=1)
2022-12-01T10:50:46.5604670Z 
2022-12-01T10:50:46.5604741Z Generating XML reports...
2022-12-01T10:50:46.5605208Z Generated XML report: test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupNCCLWrapperTest-20221201105046.xml
2022-12-01T10:50:46.5605487Z 
2022-12-01T10:50:46.5605903Z ##[endgroup]
2022-12-01T10:50:46.5606322Z FINISHED PRINTING LOG FILE of distributed/test_pg_wrapper (/var/lib/jenkins/workspace/test/test-reports/distributed-test_pg_wrapper_gqx32mdz)
2022-12-01T10:50:46.5606571Z 
2022-12-01T10:50:46.5606788Z Running distributed/elastic/timer/local_timer_example ... [2022-12-01 10:50:46.536759]
2022-12-01T10:50:46.5607339Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/elastic/timer/local_timer_example.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-12-01 10:50:46.537073]
2022-12-01T10:50:56.1092195Z 
2022-12-01T10:50:56.1092952Z Expand the folded group to see the log file of distributed/elastic/timer/local_timer_example
2022-12-01T10:50:56.1094208Z ##[group]PRINTING LOG FILE of distributed/elastic/timer/local_timer_example (/var/lib/jenkins/workspace/test/test-reports/distributed-elastic-timer-local_timer_example_cdr57360)
2022-12-01T10:50:56.1095168Z Test results will be stored in test-reports/python-unittest/distributed.elastic.timer.local_timer_example
2022-12-01T10:50:56.1095400Z 
2022-12-01T10:50:56.1095517Z Running tests...
2022-12-01T10:50:56.1095821Z ----------------------------------------------------------------------
2022-12-01T10:50:56.1096491Z   test_example_start_method_spawn (__main__.LocalTimerExample) ... [INFO] 2022-12-01 10:50:47,732 api: Starting LocalTimerServer... max_interval=0.01, daemon=True
2022-12-01T10:50:56.1097035Z [INFO] 2022-12-01 10:50:47,732 api: Starting watchdog thread...
2022-12-01T10:50:56.1097502Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:50:56.1097852Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:50:56.1098344Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:50:56.1098702Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:50:56.1099114Z [INFO] 2022-12-01 10:50:49,348 api: Timer client configured to: LocalTimerClient
2022-12-01T10:50:56.1099556Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:50:56.1099936Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:50:56.1100569Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:50:56.1101171Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:50:56.1101715Z [INFO] 2022-12-01 10:50:49,351 api: Timer client configured to: LocalTimerClient
2022-12-01T10:50:56.1102221Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:50:56.1102541Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:50:56.1103035Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:50:56.1103386Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:50:56.1103809Z [INFO] 2022-12-01 10:50:49,355 api: Timer client configured to: LocalTimerClient
2022-12-01T10:50:56.1104278Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:50:56.1104674Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:50:56.1105106Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:50:56.1105508Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:50:56.1105879Z [INFO] 2022-12-01 10:50:49,382 api: Timer client configured to: LocalTimerClient
2022-12-01T10:50:56.1106372Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:50:56.1106706Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:50:56.1107183Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:50:56.1107546Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:50:56.1107959Z [INFO] 2022-12-01 10:50:49,402 api: Timer client configured to: LocalTimerClient
2022-12-01T10:50:56.1108439Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:50:56.1108777Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:50:56.1109268Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:50:56.1109619Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:50:56.1110028Z [INFO] 2022-12-01 10:50:49,414 api: Timer client configured to: LocalTimerClient
2022-12-01T10:50:56.1110462Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:50:56.1110846Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:50:56.1111263Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:50:56.1111669Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:50:56.1112036Z [INFO] 2022-12-01 10:50:49,446 api: Timer client configured to: LocalTimerClient
2022-12-01T10:50:56.1112538Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:50:56.1112860Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:50:56.1113356Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:50:56.1113703Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:50:56.1114134Z [INFO] 2022-12-01 10:50:49,461 api: Timer client configured to: LocalTimerClient
2022-12-01T10:50:56.1114576Z [INFO] 2022-12-01 10:50:50,450 api: Reaping worker_id=[13907]. Expired timers: ['/opt/conda/lib/python3.7/contextlib.py#112']
2022-12-01T10:50:56.1114992Z [INFO] 2022-12-01 10:50:50,450 api: Successfully reaped worker=[13907]
2022-12-01T10:50:56.1115531Z [INFO] 2022-12-01 10:50:50,480 api: Reaping worker_id=[13911]. Expired timers: ['/opt/conda/lib/python3.7/contextlib.py#112']
2022-12-01T10:50:56.1115947Z [INFO] 2022-12-01 10:50:50,480 api: Successfully reaped worker=[13911]
2022-12-01T10:50:56.1116435Z [INFO] 2022-12-01 10:50:50,521 api: Reaping worker_id=[13913]. Expired timers: ['/opt/conda/lib/python3.7/contextlib.py#112']
2022-12-01T10:50:56.1116812Z [INFO] 2022-12-01 10:50:50,521 api: Successfully reaped worker=[13913]
2022-12-01T10:50:56.1117299Z [INFO] 2022-12-01 10:50:50,521 api: Reaping worker_id=[13909]. Expired timers: ['/opt/conda/lib/python3.7/contextlib.py#112']
2022-12-01T10:50:56.1117697Z [INFO] 2022-12-01 10:50:50,521 api: Successfully reaped worker=[13909]
2022-12-01T10:50:56.1118044Z [INFO] 2022-12-01 10:50:50,527 api: Stopping LocalTimerServer
2022-12-01T10:50:56.1118363Z [INFO] 2022-12-01 10:50:50,527 api: Stopping watchdog thread...
2022-12-01T10:50:56.1118622Z ok (3.040s)
2022-12-01T10:50:56.1119055Z   test_torch_mp_example (__main__.LocalTimerExample) ... [INFO] 2022-12-01 10:50:50,534 api: Starting LocalTimerServer... max_interval=0.01, daemon=True
2022-12-01T10:50:56.1119514Z [INFO] 2022-12-01 10:50:50,534 api: Starting watchdog thread...
2022-12-01T10:50:56.1119932Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:50:56.1120312Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:50:56.1120762Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:50:56.1121155Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:50:56.1121513Z [INFO] 2022-12-01 10:50:52,119 api: Timer client configured to: LocalTimerClient
2022-12-01T10:50:56.1122005Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:50:56.1122359Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:50:56.1122817Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:50:56.1123212Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:50:56.1123581Z [INFO] 2022-12-01 10:50:52,140 api: Timer client configured to: LocalTimerClient
2022-12-01T10:50:56.1124073Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:50:56.1124400Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:50:56.1124871Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:50:56.1125221Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:50:56.1125648Z [INFO] 2022-12-01 10:50:52,142 api: Timer client configured to: LocalTimerClient
2022-12-01T10:50:56.1126070Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:50:56.1126460Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:50:56.1126884Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:50:56.1127292Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:50:56.1127639Z [INFO] 2022-12-01 10:50:52,144 api: Timer client configured to: LocalTimerClient
2022-12-01T10:50:56.1128129Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:50:56.1128459Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:50:56.1128928Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:50:56.1129274Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:50:56.1129687Z [INFO] 2022-12-01 10:50:52,147 api: Timer client configured to: LocalTimerClient
2022-12-01T10:50:56.1130201Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:50:56.1130522Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:50:56.1130954Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:50:56.1131301Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:50:56.1131695Z [INFO] 2022-12-01 10:50:52,148 api: Timer client configured to: LocalTimerClient
2022-12-01T10:50:56.1132242Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:50:56.1132646Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:50:56.1133085Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:50:56.1133419Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:50:56.1133780Z [INFO] 2022-12-01 10:50:52,150 api: Timer client configured to: LocalTimerClient
2022-12-01T10:50:56.1134213Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:50:56.1134543Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:50:56.1134954Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:50:56.1135298Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:50:56.1135662Z [INFO] 2022-12-01 10:50:52,156 api: Timer client configured to: LocalTimerClient
2022-12-01T10:50:56.1136105Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:50:56.1136469Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:50:56.1136903Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:50:56.1137248Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:50:56.1137593Z [INFO] 2022-12-01 10:50:54,665 api: Timer client configured to: LocalTimerClient
2022-12-01T10:50:56.1138030Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:50:56.1138359Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:50:56.1138765Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:50:56.1139109Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:50:56.1139479Z [INFO] 2022-12-01 10:50:54,669 api: Timer client configured to: LocalTimerClient
2022-12-01T10:50:56.1139911Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:50:56.1140224Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:50:56.1140751Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:50:56.1141097Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:50:56.1141460Z [INFO] 2022-12-01 10:50:54,718 api: Timer client configured to: LocalTimerClient
2022-12-01T10:50:56.1141877Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:50:56.1142204Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:50:56.1142633Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:50:56.1143066Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:50:56.1143528Z [INFO] 2022-12-01 10:50:54,725 api: Timer client configured to: LocalTimerClient
2022-12-01T10:50:56.1143964Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:50:56.1144293Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:50:56.1144752Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:50:56.1145193Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:50:56.1145558Z [INFO] 2022-12-01 10:50:54,746 api: Timer client configured to: LocalTimerClient
2022-12-01T10:50:56.1146023Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:50:56.1146353Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:50:56.1146784Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:50:56.1147132Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:50:56.1147480Z [INFO] 2022-12-01 10:50:54,746 api: Timer client configured to: LocalTimerClient
2022-12-01T10:50:56.1147911Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:50:56.1148243Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:50:56.1148654Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:50:56.1149026Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:50:56.1149388Z [INFO] 2022-12-01 10:50:54,768 api: Timer client configured to: LocalTimerClient
2022-12-01T10:50:56.1149821Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:50:56.1150138Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:50:56.1150565Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:50:56.1150910Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:50:56.1151271Z [INFO] 2022-12-01 10:50:54,769 api: Timer client configured to: LocalTimerClient
2022-12-01T10:50:56.1151689Z [INFO] 2022-12-01 10:50:55,761 api: Reaping worker_id=[14089]. Expired timers: ['/opt/conda/lib/python3.7/contextlib.py#112']
2022-12-01T10:50:56.1152070Z [INFO] 2022-12-01 10:50:55,761 api: Successfully reaped worker=[14089]
2022-12-01T10:50:56.1152489Z [INFO] 2022-12-01 10:50:55,761 api: Reaping worker_id=[14086]. Expired timers: ['/opt/conda/lib/python3.7/contextlib.py#112']
2022-12-01T10:50:56.1152855Z [INFO] 2022-12-01 10:50:55,761 api: Successfully reaped worker=[14086]
2022-12-01T10:50:56.1153179Z [INFO] 2022-12-01 10:50:55,798 api: Stopping LocalTimerServer
2022-12-01T10:50:56.1153498Z [INFO] 2022-12-01 10:50:55,798 api: Stopping watchdog thread...
2022-12-01T10:50:56.1153708Z ok (5.270s)
2022-12-01T10:50:56.1153802Z 
2022-12-01T10:50:56.1154003Z ----------------------------------------------------------------------
2022-12-01T10:50:56.1154252Z Ran 2 tests in 8.311s
2022-12-01T10:50:56.1154366Z 
2022-12-01T10:50:56.1154429Z OK
2022-12-01T10:50:56.1154521Z 
2022-12-01T10:50:56.1154593Z Generating XML reports...
2022-12-01T10:50:56.1155072Z Generated XML report: test-reports/python-unittest/distributed.elastic.timer.local_timer_example/TEST-LocalTimerExample-20221201105047.xml
2022-12-01T10:50:56.1155354Z 
2022-12-01T10:50:56.1155683Z ##[endgroup]
2022-12-01T10:50:56.1156176Z FINISHED PRINTING LOG FILE of distributed/elastic/timer/local_timer_example (/var/lib/jenkins/workspace/test/test-reports/distributed-elastic-timer-local_timer_example_cdr57360)
2022-12-01T10:50:56.1156561Z 
2022-12-01T10:50:56.1156841Z Running distributed/algorithms/quantization/test_quantization ... [2022-12-01 10:50:56.109545]
2022-12-01T10:50:56.1157235Z MPI not available -- MPI backend tests will be skipped
2022-12-01T10:50:56.1196059Z Running distributed tests for the test backend with env init_method
2022-12-01T10:50:56.1211738Z Running distributed tests for the gloo backend with env init_method
2022-12-01T10:50:56.1223088Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/algorithms/quantization/test_quantization.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-12-01 10:50:56.121997]
2022-12-01T10:50:56.1229048Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/algorithms/quantization/test_quantization.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-12-01 10:50:56.122683]
2022-12-01T10:50:57.2923413Z 
2022-12-01T10:50:57.2924057Z Expand the folded group to see the log file of distributed/algorithms/quantization/test_quantization
2022-12-01T10:50:57.2925433Z ##[group]PRINTING LOG FILE of distributed/algorithms/quantization/test_quantization (/var/lib/jenkins/workspace/test/test-reports/distributed-algorithms-quantization-test_quantization_omg79zn9)
2022-12-01T10:50:57.2926216Z <unittest.suite.TestSuite tests=[]>
2022-12-01T10:50:57.2926465Z 
2022-12-01T10:50:57.2926845Z ##[endgroup]
2022-12-01T10:50:57.2927978Z FINISHED PRINTING LOG FILE of distributed/algorithms/quantization/test_quantization (/var/lib/jenkins/workspace/test/test-reports/distributed-algorithms-quantization-test_quantization_omg79zn9)
2022-12-01T10:50:57.2928576Z 
2022-12-01T10:51:07.5559244Z 
2022-12-01T10:51:07.5560026Z Expand the folded group to see the log file of distributed/algorithms/quantization/test_quantization
2022-12-01T10:51:07.5561366Z ##[group]PRINTING LOG FILE of distributed/algorithms/quantization/test_quantization (/var/lib/jenkins/workspace/test/test-reports/distributed-algorithms-quantization-test_quantization_vrjvz4c1)
2022-12-01T10:51:07.5562628Z <unittest.suite.TestSuite tests=[<__main__.DistQuantizationTests testMethod=test_all_gather_bfp16>, <__main__.DistQuantizationTests testMethod=test_all_gather_fp16>, <__main__.DistQuantizationTests testMethod=test_all_to_all_bfp16>, <__main__.DistQuantizationTests testMethod=test_all_to_all_fp16>, <__main__.DistQuantizationTests testMethod=test_all_to_all_single_bfp16>, <__main__.DistQuantizationTests testMethod=test_all_to_all_single_fp16>]>
2022-12-01T10:51:07.5563280Z test_all_gather_bfp16 (__main__.DistQuantizationTests)
2022-12-01T10:51:07.5563556Z test_all_gather_fp16 (__main__.DistQuantizationTests)
2022-12-01T10:51:07.5563811Z test_all_to_all_bfp16 (__main__.DistQuantizationTests)
2022-12-01T10:51:07.5564075Z test_all_to_all_fp16 (__main__.DistQuantizationTests)
2022-12-01T10:51:07.5564346Z test_all_to_all_single_bfp16 (__main__.DistQuantizationTests)
2022-12-01T10:51:07.5564612Z test_all_to_all_single_fp16 (__main__.DistQuantizationTests)
2022-12-01T10:51:07.5564867Z <unittest.suite.TestSuite tests=[]>
2022-12-01T10:51:07.5565397Z Test results will be stored in test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization
2022-12-01T10:51:07.5565911Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:51:07.5566244Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:51:07.5566675Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:51:07.5567027Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:51:07.5567197Z 
2022-12-01T10:51:07.5567258Z Running tests...
2022-12-01T10:51:07.5567561Z ----------------------------------------------------------------------
2022-12-01T10:51:07.5567948Z   test_all_gather_bfp16 (__main__.DistQuantizationTests) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14209
2022-12-01T10:51:07.5568559Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14210
2022-12-01T10:51:07.5568959Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14211
2022-12-01T10:51:07.5569413Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:51:07.5569749Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:51:07.5570167Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:51:07.5570512Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:51:07.5570839Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:51:07.5571300Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:51:07.5571618Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:51:07.5572120Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:51:07.5572470Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:51:07.5572780Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:51:07.5573245Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:51:07.5573574Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:51:07.5574004Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:51:07.5574335Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:51:07.5574656Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:51:07.5575017Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:51:07.5575375Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:51:07.5575736Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:51:07.5576229Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:51:07.5576749Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:51:07.5577251Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:51:07.5577536Z ok (1.551s)
2022-12-01T10:51:07.5577641Z 
2022-12-01T10:51:07.5577839Z ----------------------------------------------------------------------
2022-12-01T10:51:07.5578082Z Ran 1 test in 1.551s
2022-12-01T10:51:07.5578182Z 
2022-12-01T10:51:07.5578242Z OK
2022-12-01T10:51:07.5578336Z 
2022-12-01T10:51:07.5578419Z Generating XML reports...
2022-12-01T10:51:07.5578926Z Generated XML report: test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221201105058.xml
2022-12-01T10:51:07.5579501Z Test results will be stored in test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization
2022-12-01T10:51:07.5580006Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:51:07.5580333Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:51:07.5580921Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:51:07.5581254Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:51:07.5581423Z 
2022-12-01T10:51:07.5581576Z Running tests...
2022-12-01T10:51:07.5581882Z ----------------------------------------------------------------------
2022-12-01T10:51:07.5582307Z   test_all_gather_fp16 (__main__.DistQuantizationTests) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14262
2022-12-01T10:51:07.5582696Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14263
2022-12-01T10:51:07.5583024Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14264
2022-12-01T10:51:07.5583476Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:51:07.5583792Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:51:07.5584220Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:51:07.5584564Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:51:07.5584890Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:51:07.5585337Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:51:07.5585661Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:51:07.5586124Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:51:07.5586450Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:51:07.5586771Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:51:07.5587225Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:51:07.5587546Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:51:07.5587961Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:51:07.5588299Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:51:07.5588619Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:51:07.5588967Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:51:07.5589335Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:51:07.5589694Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:51:07.5590184Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:51:07.5590689Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:51:07.5591204Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:51:07.5591493Z ok (1.652s)
2022-12-01T10:51:07.5591595Z 
2022-12-01T10:51:07.5591793Z ----------------------------------------------------------------------
2022-12-01T10:51:07.5592020Z Ran 1 test in 1.653s
2022-12-01T10:51:07.5592132Z 
2022-12-01T10:51:07.5592191Z OK
2022-12-01T10:51:07.5592281Z 
2022-12-01T10:51:07.5592364Z Generating XML reports...
2022-12-01T10:51:07.5592851Z Generated XML report: test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221201105100.xml
2022-12-01T10:51:07.5593435Z Test results will be stored in test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization
2022-12-01T10:51:07.5593938Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:51:07.5594270Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:51:07.5594732Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:51:07.5595115Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:51:07.5595280Z 
2022-12-01T10:51:07.5595354Z Running tests...
2022-12-01T10:51:07.5595645Z ----------------------------------------------------------------------
2022-12-01T10:51:07.5596000Z   test_all_to_all_bfp16 (__main__.DistQuantizationTests) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-12-01T10:51:07.5596211Z 
2022-12-01T10:51:07.5596402Z ----------------------------------------------------------------------
2022-12-01T10:51:07.5596639Z Ran 1 test in 0.001s
2022-12-01T10:51:07.5596751Z 
2022-12-01T10:51:07.5596810Z OK (skipped=1)
2022-12-01T10:51:07.5596915Z 
2022-12-01T10:51:07.5596999Z Generating XML reports...
2022-12-01T10:51:07.5597500Z Generated XML report: test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221201105103.xml
2022-12-01T10:51:07.5598082Z Test results will be stored in test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization
2022-12-01T10:51:07.5598573Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:51:07.5598901Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:51:07.5599325Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:51:07.5599654Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:51:07.5599819Z 
2022-12-01T10:51:07.5599890Z Running tests...
2022-12-01T10:51:07.5600186Z ----------------------------------------------------------------------
2022-12-01T10:51:07.5600536Z   test_all_to_all_fp16 (__main__.DistQuantizationTests) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-12-01T10:51:07.5600742Z 
2022-12-01T10:51:07.5600924Z ----------------------------------------------------------------------
2022-12-01T10:51:07.5601180Z Ran 1 test in 0.001s
2022-12-01T10:51:07.5601290Z 
2022-12-01T10:51:07.5601361Z OK (skipped=1)
2022-12-01T10:51:07.5601466Z 
2022-12-01T10:51:07.5601536Z Generating XML reports...
2022-12-01T10:51:07.5602034Z Generated XML report: test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221201105104.xml
2022-12-01T10:51:07.5602617Z Test results will be stored in test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization
2022-12-01T10:51:07.5603115Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:51:07.5603430Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:51:07.5603861Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:51:07.5604209Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:51:07.5604375Z 
2022-12-01T10:51:07.5604446Z Running tests...
2022-12-01T10:51:07.5604732Z ----------------------------------------------------------------------
2022-12-01T10:51:07.5605091Z   test_all_to_all_single_bfp16 (__main__.DistQuantizationTests) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-12-01T10:51:07.5605302Z 
2022-12-01T10:51:07.5605490Z ----------------------------------------------------------------------
2022-12-01T10:51:07.5605712Z Ran 1 test in 0.001s
2022-12-01T10:51:07.5605824Z 
2022-12-01T10:51:07.5605893Z OK (skipped=1)
2022-12-01T10:51:07.5605998Z 
2022-12-01T10:51:07.5606080Z Generating XML reports...
2022-12-01T10:51:07.5606576Z Generated XML report: test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221201105105.xml
2022-12-01T10:51:07.5607185Z Test results will be stored in test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization
2022-12-01T10:51:07.5607693Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:51:07.5608057Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:51:07.5608488Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:51:07.5608818Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:51:07.5608983Z 
2022-12-01T10:51:07.5609055Z Running tests...
2022-12-01T10:51:07.5609353Z ----------------------------------------------------------------------
2022-12-01T10:51:07.5609698Z   test_all_to_all_single_fp16 (__main__.DistQuantizationTests) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-12-01T10:51:07.5609908Z 
2022-12-01T10:51:07.5610098Z ----------------------------------------------------------------------
2022-12-01T10:51:07.5610336Z Ran 1 test in 0.001s
2022-12-01T10:51:07.5610447Z 
2022-12-01T10:51:07.5610520Z OK (skipped=1)
2022-12-01T10:51:07.5610615Z 
2022-12-01T10:51:07.5610697Z Generating XML reports...
2022-12-01T10:51:07.5611193Z Generated XML report: test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221201105107.xml
2022-12-01T10:51:07.5611491Z 
2022-12-01T10:51:07.5611787Z ##[endgroup]
2022-12-01T10:51:07.5612437Z FINISHED PRINTING LOG FILE of distributed/algorithms/quantization/test_quantization (/var/lib/jenkins/workspace/test/test-reports/distributed-algorithms-quantization-test_quantization_vrjvz4c1)
2022-12-01T10:51:07.5612767Z 
2022-12-01T10:51:07.6445257Z Running distributed tests for the test backend with file init_method
2022-12-01T10:51:07.6458158Z Running distributed tests for the gloo backend with file init_method
2022-12-01T10:51:07.6470114Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/algorithms/quantization/test_quantization.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-12-01 10:51:07.646714]
2022-12-01T10:51:07.6481179Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/algorithms/quantization/test_quantization.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-12-01 10:51:07.647874]
2022-12-01T10:51:08.8207137Z 
2022-12-01T10:51:08.8207888Z Expand the folded group to see the log file of distributed/algorithms/quantization/test_quantization
2022-12-01T10:51:08.8208811Z ##[group]PRINTING LOG FILE of distributed/algorithms/quantization/test_quantization (/var/lib/jenkins/workspace/test/test-reports/distributed-algorithms-quantization-test_quantization_hooub2ug)
2022-12-01T10:51:08.8209247Z <unittest.suite.TestSuite tests=[]>
2022-12-01T10:51:08.8209388Z 
2022-12-01T10:51:08.8209604Z ##[endgroup]
2022-12-01T10:51:08.8210267Z FINISHED PRINTING LOG FILE of distributed/algorithms/quantization/test_quantization (/var/lib/jenkins/workspace/test/test-reports/distributed-algorithms-quantization-test_quantization_hooub2ug)
2022-12-01T10:51:08.8210592Z 
2022-12-01T10:51:19.0556271Z 
2022-12-01T10:51:19.0556842Z Expand the folded group to see the log file of distributed/algorithms/quantization/test_quantization
2022-12-01T10:51:19.0558146Z ##[group]PRINTING LOG FILE of distributed/algorithms/quantization/test_quantization (/var/lib/jenkins/workspace/test/test-reports/distributed-algorithms-quantization-test_quantization_8zc7m8xi)
2022-12-01T10:51:19.0559063Z <unittest.suite.TestSuite tests=[<__main__.DistQuantizationTests testMethod=test_all_gather_bfp16>, <__main__.DistQuantizationTests testMethod=test_all_gather_fp16>, <__main__.DistQuantizationTests testMethod=test_all_to_all_bfp16>, <__main__.DistQuantizationTests testMethod=test_all_to_all_fp16>, <__main__.DistQuantizationTests testMethod=test_all_to_all_single_bfp16>, <__main__.DistQuantizationTests testMethod=test_all_to_all_single_fp16>]>
2022-12-01T10:51:19.0559715Z test_all_gather_bfp16 (__main__.DistQuantizationTests)
2022-12-01T10:51:19.0560188Z test_all_gather_fp16 (__main__.DistQuantizationTests)
2022-12-01T10:51:19.0560523Z test_all_to_all_bfp16 (__main__.DistQuantizationTests)
2022-12-01T10:51:19.0560774Z test_all_to_all_fp16 (__main__.DistQuantizationTests)
2022-12-01T10:51:19.0561049Z test_all_to_all_single_bfp16 (__main__.DistQuantizationTests)
2022-12-01T10:51:19.0561331Z test_all_to_all_single_fp16 (__main__.DistQuantizationTests)
2022-12-01T10:51:19.0561587Z <unittest.suite.TestSuite tests=[]>
2022-12-01T10:51:19.0562105Z Test results will be stored in test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization
2022-12-01T10:51:19.0562636Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:51:19.0562975Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:51:19.0563393Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:51:19.0563746Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:51:19.0563918Z 
2022-12-01T10:51:19.0563991Z Running tests...
2022-12-01T10:51:19.0564296Z ----------------------------------------------------------------------
2022-12-01T10:51:19.0564674Z   test_all_gather_bfp16 (__main__.DistQuantizationTests) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14380
2022-12-01T10:51:19.0565064Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14381
2022-12-01T10:51:19.0565390Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14382
2022-12-01T10:51:19.0565832Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:51:19.0566163Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:51:19.0566594Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:51:19.0566940Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:51:19.0567256Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:51:19.0567715Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:51:19.0568042Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:51:19.0568471Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:51:19.0568804Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:51:19.0569127Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:51:19.0569584Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:51:19.0569901Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:51:19.0570329Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:51:19.0570671Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:51:19.0570994Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:51:19.0571345Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:51:19.0571712Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:51:19.0572177Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:51:19.0572666Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:51:19.0573245Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:51:19.0573800Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:51:19.0574094Z ok (1.653s)
2022-12-01T10:51:19.0574198Z 
2022-12-01T10:51:19.0574383Z ----------------------------------------------------------------------
2022-12-01T10:51:19.0574623Z Ran 1 test in 1.653s
2022-12-01T10:51:19.0574737Z 
2022-12-01T10:51:19.0574802Z OK
2022-12-01T10:51:19.0574893Z 
2022-12-01T10:51:19.0574977Z Generating XML reports...
2022-12-01T10:51:19.0575466Z Generated XML report: test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221201105109.xml
2022-12-01T10:51:19.0576056Z Test results will be stored in test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization
2022-12-01T10:51:19.0576563Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:51:19.0576883Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:51:19.0577311Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:51:19.0577662Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:51:19.0577827Z 
2022-12-01T10:51:19.0577899Z Running tests...
2022-12-01T10:51:19.0578183Z ----------------------------------------------------------------------
2022-12-01T10:51:19.0578575Z   test_all_gather_fp16 (__main__.DistQuantizationTests) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14433
2022-12-01T10:51:19.0578968Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14434
2022-12-01T10:51:19.0579291Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14435
2022-12-01T10:51:19.0579748Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:51:19.0580082Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:51:19.0580753Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:51:19.0581092Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:51:19.0581416Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:51:19.0581884Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:51:19.0582211Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:51:19.0582621Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:51:19.0582965Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:51:19.0583290Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:51:19.0583741Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:51:19.0584065Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:51:19.0584491Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:51:19.0584832Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:51:19.0585141Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:51:19.0585507Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:51:19.0585875Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:51:19.0586306Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:51:19.0586844Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:51:19.0587363Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:51:19.0587887Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
2022-12-01T10:51:19.0588161Z ok (1.551s)
2022-12-01T10:51:19.0588263Z 
2022-12-01T10:51:19.0588464Z ----------------------------------------------------------------------
2022-12-01T10:51:19.0588707Z Ran 1 test in 1.552s
2022-12-01T10:51:19.0588820Z 
2022-12-01T10:51:19.0588882Z OK
2022-12-01T10:51:19.0588961Z 
2022-12-01T10:51:19.0589046Z Generating XML reports...
2022-12-01T10:51:19.0589551Z Generated XML report: test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221201105112.xml
2022-12-01T10:51:19.0590144Z Test results will be stored in test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization
2022-12-01T10:51:19.0590638Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:51:19.0590968Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:51:19.0591401Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:51:19.0591745Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:51:19.0591908Z 
2022-12-01T10:51:19.0591968Z Running tests...
2022-12-01T10:51:19.0592277Z ----------------------------------------------------------------------
2022-12-01T10:51:19.0592629Z   test_all_to_all_bfp16 (__main__.DistQuantizationTests) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-12-01T10:51:19.0592840Z 
2022-12-01T10:51:19.0593032Z ----------------------------------------------------------------------
2022-12-01T10:51:19.0593258Z Ran 1 test in 0.001s
2022-12-01T10:51:19.0593370Z 
2022-12-01T10:51:19.0593440Z OK (skipped=1)
2022-12-01T10:51:19.0593547Z 
2022-12-01T10:51:19.0593631Z Generating XML reports...
2022-12-01T10:51:19.0594116Z Generated XML report: test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221201105115.xml
2022-12-01T10:51:19.0594699Z Test results will be stored in test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization
2022-12-01T10:51:19.0595201Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:51:19.0595528Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:51:19.0595942Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:51:19.0596288Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:51:19.0596454Z 
2022-12-01T10:51:19.0596525Z Running tests...
2022-12-01T10:51:19.0596814Z ----------------------------------------------------------------------
2022-12-01T10:51:19.0597167Z   test_all_to_all_fp16 (__main__.DistQuantizationTests) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-12-01T10:51:19.0597377Z 
2022-12-01T10:51:19.0597570Z ----------------------------------------------------------------------
2022-12-01T10:51:19.0597809Z Ran 1 test in 0.001s
2022-12-01T10:51:19.0597909Z 
2022-12-01T10:51:19.0597980Z OK (skipped=1)
2022-12-01T10:51:19.0598086Z 
2022-12-01T10:51:19.0598169Z Generating XML reports...
2022-12-01T10:51:19.0598668Z Generated XML report: test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221201105116.xml
2022-12-01T10:51:19.0599284Z Test results will be stored in test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization
2022-12-01T10:51:19.0599823Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:51:19.0600153Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:51:19.0600579Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:51:19.0600911Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:51:19.0601075Z 
2022-12-01T10:51:19.0601147Z Running tests...
2022-12-01T10:51:19.0601445Z ----------------------------------------------------------------------
2022-12-01T10:51:19.0601802Z   test_all_to_all_single_bfp16 (__main__.DistQuantizationTests) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-12-01T10:51:19.0602004Z 
2022-12-01T10:51:19.0602195Z ----------------------------------------------------------------------
2022-12-01T10:51:19.0602435Z Ran 1 test in 0.001s
2022-12-01T10:51:19.0602548Z 
2022-12-01T10:51:19.0602621Z OK (skipped=1)
2022-12-01T10:51:19.0602727Z 
2022-12-01T10:51:19.0602797Z Generating XML reports...
2022-12-01T10:51:19.0603297Z Generated XML report: test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221201105117.xml
2022-12-01T10:51:19.0603885Z Test results will be stored in test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization
2022-12-01T10:51:19.0604394Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:51:19.0604715Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:51:19.0605141Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:51:19.0605484Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:51:19.0605645Z 
2022-12-01T10:51:19.0605720Z Running tests...
2022-12-01T10:51:19.0606007Z ----------------------------------------------------------------------
2022-12-01T10:51:19.0606367Z   test_all_to_all_single_fp16 (__main__.DistQuantizationTests) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-12-01T10:51:19.0606577Z 
2022-12-01T10:51:19.0606768Z ----------------------------------------------------------------------
2022-12-01T10:51:19.0606992Z Ran 1 test in 0.001s
2022-12-01T10:51:19.0607104Z 
2022-12-01T10:51:19.0607175Z OK (skipped=1)
2022-12-01T10:51:19.0607280Z 
2022-12-01T10:51:19.0607363Z Generating XML reports...
2022-12-01T10:51:19.0607861Z Generated XML report: test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221201105118.xml
2022-12-01T10:51:19.0608148Z 
2022-12-01T10:51:19.0608441Z ##[endgroup]
2022-12-01T10:51:19.0608982Z FINISHED PRINTING LOG FILE of distributed/algorithms/quantization/test_quantization (/var/lib/jenkins/workspace/test/test-reports/distributed-algorithms-quantization-test_quantization_8zc7m8xi)
2022-12-01T10:51:19.0609311Z 
2022-12-01T10:51:19.0642775Z Running distributed/_shard/sharded_tensor/test_sharded_tensor ... [2022-12-01 10:51:19.064001]
2022-12-01T10:51:19.0645119Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/sharded_tensor/test_sharded_tensor.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-12-01 10:51:19.064277]
2022-12-01T10:51:23.5427388Z 
2022-12-01T10:51:23.5428425Z Expand the folded group to see the log file of distributed/_shard/sharded_tensor/test_sharded_tensor
2022-12-01T10:51:23.5429382Z ##[group]PRINTING LOG FILE of distributed/_shard/sharded_tensor/test_sharded_tensor (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharded_tensor-test_sharded_tensor_c2o0akbw)
2022-12-01T10:51:23.5430393Z Test results will be stored in test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor
2022-12-01T10:51:23.5430750Z 
2022-12-01T10:51:23.5431033Z Running tests...
2022-12-01T10:51:23.5431479Z ----------------------------------------------------------------------
2022-12-01T10:51:23.5432211Z   test_empty (__main__.TestCreateTensorFromParams) ... skip: CUDA GPU is needed (0.001s)
2022-12-01T10:51:23.5432607Z   test_local_tensor (__main__.TestLocalTensor) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-12-01T10:51:23.5432978Z   test_local_tensor_error (__main__.TestLocalTensor) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-12-01T10:51:23.5433350Z   test_collect_local_shard (__main__.TestModuleHookApi) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-12-01T10:51:23.5433794Z   test_reshard_output (__main__.TestModuleHookApi) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-12-01T10:51:23.5434266Z   test_create_shard_with_no_placement (__main__.TestShardMetadata) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2022-12-01T10:51:23.5463844Z   test_shard_metadata_init (__main__.TestShardMetadata) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-12-01T10:51:23.5480256Z   test_shard_parameter (__main__.TestShardParameter) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-12-01T10:51:23.5480989Z   test_shard_parameter_errors (__main__.TestShardParameter) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-12-01T10:51:23.5481497Z   test_shard_tensor (__main__.TestShardTensor) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-12-01T10:51:23.5481882Z   test_shard_tensor_errors (__main__.TestShardTensor) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-12-01T10:51:23.5482437Z   test_cleanup (__main__.TestShardedTensorChunked) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-12-01T10:51:23.5483136Z   test_complete_world_size (__main__.TestShardedTensorChunked) ... skip: c10d was not compiled with the NCCL backend (0.002s)
2022-12-01T10:51:23.5483818Z   test_create_sharded_tensor_like (__main__.TestShardedTensorChunked)
2022-12-01T10:51:23.5484520Z Test tensor like methods, i.e. torch.zeros_like(...), torch.full_like, etc. ... skip: c10d was not compiled with the NCCL backend (0.002s)
2022-12-01T10:51:23.5484936Z   test_create_sharded_tensor_with_full (__main__.TestShardedTensorChunked)
2022-12-01T10:51:23.5485261Z Test sharded_tensor.full(...) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-12-01T10:51:23.5485588Z   test_create_sharded_tensor_with_ones (__main__.TestShardedTensorChunked)
2022-12-01T10:51:23.5485921Z Test sharded_tensor.ones(...) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-12-01T10:51:23.5486326Z   test_create_sharded_tensor_with_rand (__main__.TestShardedTensorChunked)
2022-12-01T10:51:23.5486781Z Test sharded_tensor.rand(...)/randn(...) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-12-01T10:51:23.5487398Z   test_create_sharded_tensor_with_zeros (__main__.TestShardedTensorChunked)
2022-12-01T10:51:23.5487907Z Test sharded_tensor.zeros(...) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-12-01T10:51:23.5488368Z   test_gather_even (__main__.TestShardedTensorChunked)
2022-12-01T10:51:23.5488969Z Test _sharded_tensor.gather(...) with evenly distributed._shards ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-12-01T10:51:23.5489556Z   test_gather_uneven (__main__.TestShardedTensorChunked)
2022-12-01T10:51:23.5490137Z Test _sharded_tensor.gather(...) with unevenly distributed._shards ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-12-01T10:51:23.5490814Z   test_insufficient_sharding_dims (__main__.TestShardedTensorChunked) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-12-01T10:51:23.5491449Z   test_invalid_pg_rpc_ranks (__main__.TestShardedTensorChunked) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-12-01T10:51:23.5492492Z   test_invalid_sharding (__main__.TestShardedTensorChunked) ... skip: c10d was not compiled with the NCCL backend (0.002s)
2022-12-01T10:51:23.5493341Z   test_load_state_dict_errors (__main__.TestShardedTensorChunked) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-12-01T10:51:23.5494112Z   test_multiple_local_shards (__main__.TestShardedTensorChunked) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-12-01T10:51:23.5494736Z   test_new_group (__main__.TestShardedTensorChunked) ... skip: c10d was not compiled with the NCCL backend (0.002s)
2022-12-01T10:51:23.5495134Z   test_partial_world_size (__main__.TestShardedTensorChunked) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-12-01T10:51:23.5495547Z   test_sharded_tensor_metadata (__main__.TestShardedTensorChunked) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-12-01T10:51:23.5495966Z   test_sharded_tensor_sizes (__main__.TestShardedTensorChunked) ... skip: c10d was not compiled with the NCCL backend (0.002s)
2022-12-01T10:51:23.5496364Z   test_sharding_columns (__main__.TestShardedTensorChunked) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-12-01T10:51:23.5496768Z   test_state_dict (__main__.TestShardedTensorChunked) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-12-01T10:51:23.5497168Z   test_state_dict_new_group (__main__.TestShardedTensorChunked) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-12-01T10:51:23.5497585Z   test_state_dict_no_sharded_tensors (__main__.TestShardedTensorChunked) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-12-01T10:51:23.5497984Z   test_custom_op (__main__.TestShardedTensorCustomOps) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-12-01T10:51:23.5498390Z   test_custom_op_errors (__main__.TestShardedTensorCustomOps) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2022-12-01T10:51:23.5498807Z   test_custom_op_override (__main__.TestShardedTensorCustomOps) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-12-01T10:51:23.5499171Z   test_create_sharded_tensor_with_ones (__main__.TestShardedTensorEnumerable)
2022-12-01T10:51:23.5499512Z Test sharded_tensor.ones(...) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-12-01T10:51:23.5499826Z   test_gather_even (__main__.TestShardedTensorEnumerable)
2022-12-01T10:51:23.5500182Z Test _sharded_tensor.gather(...) with evenly distributed._shards ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-12-01T10:51:23.5500755Z   test_gather_uneven (__main__.TestShardedTensorEnumerable)
2022-12-01T10:51:23.5501116Z Test _sharded_tensor.gather(...) with unevenly distributed._shards ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-12-01T10:51:23.5501534Z   test_grid_sharding (__main__.TestShardedTensorEnumerable) ... skip: c10d was not compiled with the NCCL backend (0.002s)
2022-12-01T10:51:23.5501960Z   test_multiple_local_shards (__main__.TestShardedTensorEnumerable) ... skip: c10d was not compiled with the NCCL backend (0.002s)
2022-12-01T10:51:23.5502360Z   test_new_group (__main__.TestShardedTensorEnumerable) ... skip: c10d was not compiled with the NCCL backend (0.002s)
2022-12-01T10:51:23.5502774Z   test_partial_world_size (__main__.TestShardedTensorEnumerable) ... skip: c10d was not compiled with the NCCL backend (0.002s)
2022-12-01T10:51:23.5503202Z   test_sharded_tensor_device (__main__.TestShardedTensorEnumerable) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-12-01T10:51:23.5503634Z   test_sharded_tensor_metadata (__main__.TestShardedTensorEnumerable) ... skip: c10d was not compiled with the NCCL backend (0.002s)
2022-12-01T10:51:23.5504053Z   test_sharded_tensor_to_cpu (__main__.TestShardedTensorEnumerable) ... skip: c10d was not compiled with the NCCL backend (0.002s)
2022-12-01T10:51:23.5504480Z   test_sharded_tensor_to_cuda (__main__.TestShardedTensorEnumerable) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-12-01T10:51:23.5504981Z   test_sharded_tensor_to_test (__main__.TestShardedTensorEnumerable) ... skip: c10d was not compiled with the NCCL backend (0.002s)
2022-12-01T10:51:23.5505449Z   test_uneven_shards (__main__.TestShardedTensorEnumerable) ... skip: c10d was not compiled with the NCCL backend (0.002s)
2022-12-01T10:51:23.5505849Z   test_with_rpc_names (__main__.TestShardedTensorEnumerable) ... skip: c10d was not compiled with the NCCL backend (0.002s)
2022-12-01T10:51:23.5506283Z   test_init_from_local_shards (__main__.TestShardedTensorFromLocalShards) ... skip: c10d was not compiled with the NCCL backend (0.002s)
2022-12-01T10:51:23.5506760Z   test_init_from_local_shards_and_global_metadata (__main__.TestShardedTensorFromLocalShards) ... skip: c10d was not compiled with the NCCL backend (0.002s)
2022-12-01T10:51:23.5507262Z   test_init_from_local_shards_and_global_metadata_invalid_shards (__main__.TestShardedTensorFromLocalShards) ... skip: c10d was not compiled with the NCCL backend (0.003s)
2022-12-01T10:51:23.5507754Z   test_init_from_local_shards_invalid_local_shards (__main__.TestShardedTensorFromLocalShards) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-12-01T10:51:23.5508284Z   test_init_from_local_shards_invalid_pin_memory (__main__.TestShardedTensorFromLocalShards) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14526
2022-12-01T10:51:23.5508726Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14527
2022-12-01T10:51:23.5509065Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14528
2022-12-01T10:51:23.5509389Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 14529
2022-12-01T10:51:23.5509946Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:51:23.5510285Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:51:23.5510726Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:51:23.5511065Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:51:23.5511399Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:51:23.5511867Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:51:23.5512185Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:51:23.5512615Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:51:23.5512963Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:51:23.5513286Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:51:23.5513736Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:51:23.5514068Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:51:23.5514503Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:51:23.5514835Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:51:23.5515159Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:51:23.5515619Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:51:23.5515945Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:51:23.5516358Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:51:23.5516702Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:51:23.5517069Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:51:23.5517432Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:51:23.5517846Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:51:23.5518343Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:51:23.5518743Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:51:23.5519099Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-12-01T10:51:23.5519593Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:51:23.5520112Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:51:23.5520630Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:51:23.5520942Z skip: Need at least 4 CUDA devices (1.655s)
2022-12-01T10:51:23.5521327Z   test_init_from_local_shards_invalid_property_cross_ranks (__main__.TestShardedTensorFromLocalShards) ... skip: c10d was not compiled with the NCCL backend (0.002s)
2022-12-01T10:51:23.5521833Z   test_init_from_local_shards_invalid_shards_gaps (__main__.TestShardedTensorFromLocalShards) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-12-01T10:51:23.5522326Z   test_init_from_local_shards_invalid_shards_overlap (__main__.TestShardedTensorFromLocalShards) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-12-01T10:51:23.5522799Z   test_init_from_local_shards_new_group (__main__.TestShardedTensorFromLocalShards) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-12-01T10:51:23.5523255Z   test_local_shards (__main__.TestShardedTensorFromLocalShards) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-12-01T10:51:23.5523764Z   test_st_base_init_from_local_shards_and_global_metadata (__main__.TestShardedTensorFromLocalShards) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14582
2022-12-01T10:51:23.5524208Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14583
2022-12-01T10:51:23.5524526Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14584
2022-12-01T10:51:23.5524859Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 14585
2022-12-01T10:51:23.5525320Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:51:23.5525660Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:51:23.5526087Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:51:23.5526440Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:51:23.5526770Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:51:23.5527222Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:51:23.5527552Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:51:23.5527986Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:51:23.5528334Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:51:23.5528643Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:51:23.5529106Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:51:23.5529475Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:51:23.5529910Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:51:23.5530279Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:51:23.5530599Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:51:23.5531061Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:51:23.5531378Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:51:23.5531813Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:51:23.5532246Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:51:23.5532573Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:51:23.5532895Z skip: Need at least 4 CUDA devices (1.419s)
2022-12-01T10:51:23.5533253Z   test_init_from_local_tensor (__main__.TestShardedTensorFromLocalTensor) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-12-01T10:51:23.5533720Z   test_init_from_local_tensor_errors (__main__.TestShardedTensorFromLocalTensor) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-12-01T10:51:23.5534127Z   test_serialize_and_deserialize (__main__.TestShardedTensorMetadata) ... ok (0.035s)
2022-12-01T10:51:23.5534305Z 
2022-12-01T10:51:23.5534510Z ----------------------------------------------------------------------
2022-12-01T10:51:23.5534762Z Ran 64 tests in 3.184s
2022-12-01T10:51:23.5534877Z 
2022-12-01T10:51:23.5534951Z OK (skipped=63)
2022-12-01T10:51:23.5535063Z 
2022-12-01T10:51:23.5535134Z Generating XML reports...
2022-12-01T10:51:23.5535642Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardedTensorMetadata-20221201105120.xml
2022-12-01T10:51:23.5536317Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestCreateTensorFromParams-20221201105120.xml
2022-12-01T10:51:23.5536961Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestLocalTensor-20221201105120.xml
2022-12-01T10:51:23.5537558Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestModuleHookApi-20221201105120.xml
2022-12-01T10:51:23.5538304Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardMetadata-20221201105120.xml
2022-12-01T10:51:23.5538932Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardParameter-20221201105120.xml
2022-12-01T10:51:23.5539545Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardTensor-20221201105120.xml
2022-12-01T10:51:23.5540175Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardedTensorChunked-20221201105120.xml
2022-12-01T10:51:23.5540992Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardedTensorCustomOps-20221201105120.xml
2022-12-01T10:51:23.5541662Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardedTensorEnumerable-20221201105120.xml
2022-12-01T10:51:23.5542352Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardedTensorFromLocalShards-20221201105120.xml
2022-12-01T10:51:23.5543047Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardedTensorFromLocalTensor-20221201105120.xml
2022-12-01T10:51:23.5543356Z 
2022-12-01T10:51:23.5543862Z ##[endgroup]
2022-12-01T10:51:23.5544386Z FINISHED PRINTING LOG FILE of distributed/_shard/sharded_tensor/test_sharded_tensor (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharded_tensor-test_sharded_tensor_c2o0akbw)
2022-12-01T10:51:23.5544735Z 
2022-12-01T10:51:23.5544953Z Running distributed/elastic/utils/distributed_test ... [2022-12-01 10:51:23.543296]
2022-12-01T10:51:23.5545500Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/elastic/utils/distributed_test.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-12-01 10:51:23.543725]
2022-12-01T10:51:28.1295411Z 
2022-12-01T10:51:28.1295963Z Expand the folded group to see the log file of distributed/elastic/utils/distributed_test
2022-12-01T10:51:28.1297210Z ##[group]PRINTING LOG FILE of distributed/elastic/utils/distributed_test (/var/lib/jenkins/workspace/test/test-reports/distributed-elastic-utils-distributed_test_2zxlob14)
2022-12-01T10:51:28.1298322Z Test results will be stored in test-reports/python-unittest/distributed.elastic.utils.distributed_test
2022-12-01T10:51:28.1298550Z 
2022-12-01T10:51:28.1298625Z Running tests...
2022-12-01T10:51:28.1299003Z ----------------------------------------------------------------------
2022-12-01T10:51:28.1299310Z   test_create_store_multi (__main__.DistributedUtilTest) ... ok (0.250s)
2022-12-01T10:51:28.1299707Z   test_create_store_no_port_multi (__main__.DistributedUtilTest) ... ok (0.001s)
2022-12-01T10:51:28.1300741Z   test_create_store_single_server (__main__.DistributedUtilTest) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/66207 for allplatform(s) . If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.000s)
2022-12-01T10:51:28.1301336Z   test_create_store_timeout_on_server (__main__.DistributedUtilTest) ... ok (3.020s)
2022-12-01T10:51:28.1301756Z   test_create_store_timeout_on_worker (__main__.DistributedUtilTest) ... [E socket.cpp:860] [c10d] The client socket has timed out after 1s while trying to connect to (0bf5643a9780, 0).
2022-12-01T10:51:28.1302081Z ok (0.001s)
2022-12-01T10:51:28.1302614Z   test_port_already_in_use_on_server (__main__.DistributedUtilTest) ... [W socket.cpp:426] [c10d] The server socket has failed to bind to [::]:38681 (errno: 98 - Address already in use).
2022-12-01T10:51:28.1303129Z [W socket.cpp:426] [c10d] The server socket has failed to bind to 0.0.0.0:38681 (errno: 98 - Address already in use).
2022-12-01T10:51:28.1303473Z [E socket.cpp:462] [c10d] The server socket has failed to listen on any local network address.
2022-12-01T10:51:28.1303712Z ok (0.003s)
2022-12-01T10:51:28.1304037Z   test_port_already_in_use_on_worker (__main__.DistributedUtilTest) ... [E socket.cpp:860] [c10d] The client socket has timed out after 1s while trying to connect to (0bf5643a9780, 57817).
2022-12-01T10:51:28.1304352Z ok (0.002s)
2022-12-01T10:51:28.1304455Z 
2022-12-01T10:51:28.1304657Z ----------------------------------------------------------------------
2022-12-01T10:51:28.1304887Z Ran 7 tests in 3.279s
2022-12-01T10:51:28.1305003Z 
2022-12-01T10:51:28.1305073Z OK (skipped=1)
2022-12-01T10:51:28.1305182Z 
2022-12-01T10:51:28.1305266Z Generating XML reports...
2022-12-01T10:51:28.1305751Z Generated XML report: test-reports/python-unittest/distributed.elastic.utils.distributed_test/TEST-DistributedUtilTest-20221201105124.xml
2022-12-01T10:51:28.1306038Z 
2022-12-01T10:51:28.1306264Z ##[endgroup]
2022-12-01T10:51:28.1306761Z FINISHED PRINTING LOG FILE of distributed/elastic/utils/distributed_test (/var/lib/jenkins/workspace/test/test-reports/distributed-elastic-utils-distributed_test_2zxlob14)
2022-12-01T10:51:28.1307055Z 
2022-12-01T10:51:28.1307258Z Running distributed/rpc/test_share_memory ... [2022-12-01 10:51:28.129810]
2022-12-01T10:51:28.1307998Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/rpc/test_share_memory.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-12-01 10:51:28.130187]
2022-12-01T10:51:32.3181832Z 
2022-12-01T10:51:32.3182455Z Expand the folded group to see the log file of distributed/rpc/test_share_memory
2022-12-01T10:51:32.3183718Z ##[group]PRINTING LOG FILE of distributed/rpc/test_share_memory (/var/lib/jenkins/workspace/test/test-reports/distributed-rpc-test_share_memory_478ro6dv)
2022-12-01T10:51:32.3184410Z <unittest.suite.TestSuite tests=[]>
2022-12-01T10:51:32.3184821Z <unittest.suite.TestSuite tests=[<__main__.TestRPCPickler testMethod=test_case>]>
2022-12-01T10:51:32.3185107Z test_case (__main__.TestRPCPickler)
2022-12-01T10:51:32.3185586Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_share_memory
2022-12-01T10:51:32.3186072Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:51:32.3186499Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:51:32.3186959Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:51:32.3187363Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:51:32.3187531Z 
2022-12-01T10:51:32.3187592Z Running tests...
2022-12-01T10:51:32.3187904Z ----------------------------------------------------------------------
2022-12-01T10:51:32.3188422Z   test_case (__main__.TestRPCPickler) ... /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:51:32.3188790Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:51:32.3189212Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:51:32.3189565Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:51:32.3189905Z Exception ignored in: <function StorageWeakRef.__del__ at 0x7f470454cdd0>
2022-12-01T10:51:32.3190156Z Traceback (most recent call last):
2022-12-01T10:51:32.3190592Z   File "/opt/conda/lib/python3.7/site-packages/torch/multiprocessing/reductions.py", line 38, in __del__
2022-12-01T10:51:32.3191029Z   File "/opt/conda/lib/python3.7/site-packages/torch/storage.py", line 670, in _free_weak_ref
2022-12-01T10:51:32.3191403Z AttributeError: 'NoneType' object has no attribute '_free_weak_ref'
2022-12-01T10:51:32.3191690Z Exception ignored in: <function StorageWeakRef.__del__ at 0x7f470454cdd0>
2022-12-01T10:51:32.3191948Z Traceback (most recent call last):
2022-12-01T10:51:32.3192326Z   File "/opt/conda/lib/python3.7/site-packages/torch/multiprocessing/reductions.py", line 38, in __del__
2022-12-01T10:51:32.3192740Z   File "/opt/conda/lib/python3.7/site-packages/torch/storage.py", line 670, in _free_weak_ref
2022-12-01T10:51:32.3193107Z AttributeError: 'NoneType' object has no attribute '_free_weak_ref'
2022-12-01T10:51:32.3193334Z ok (1.780s)
2022-12-01T10:51:32.3193438Z 
2022-12-01T10:51:32.3193625Z ----------------------------------------------------------------------
2022-12-01T10:51:32.3193870Z Ran 1 test in 1.780s
2022-12-01T10:51:32.3193986Z 
2022-12-01T10:51:32.3194047Z OK
2022-12-01T10:51:32.3194139Z 
2022-12-01T10:51:32.3194223Z Generating XML reports...
2022-12-01T10:51:32.3194655Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_share_memory/TEST-TestRPCPickler-20221201105130.xml
2022-12-01T10:51:32.3194907Z 
2022-12-01T10:51:32.3195146Z ##[endgroup]
2022-12-01T10:51:32.3195593Z FINISHED PRINTING LOG FILE of distributed/rpc/test_share_memory (/var/lib/jenkins/workspace/test/test-reports/distributed-rpc-test_share_memory_478ro6dv)
2022-12-01T10:51:32.3195846Z 
2022-12-01T10:51:32.3196239Z Running distributed/elastic/utils/logging_test ... [2022-12-01 10:51:32.318504]
2022-12-01T10:51:32.3196782Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/elastic/utils/logging_test.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-12-01 10:51:32.318848]
2022-12-01T10:51:33.8658139Z 
2022-12-01T10:51:33.8658713Z Expand the folded group to see the log file of distributed/elastic/utils/logging_test
2022-12-01T10:51:33.8660013Z ##[group]PRINTING LOG FILE of distributed/elastic/utils/logging_test (/var/lib/jenkins/workspace/test/test-reports/distributed-elastic-utils-logging_test_db21yxtq)
2022-12-01T10:51:33.8660860Z Test results will be stored in test-reports/python-unittest/distributed.elastic.utils.logging_test
2022-12-01T10:51:33.8661085Z 
2022-12-01T10:51:33.8661163Z Running tests...
2022-12-01T10:51:33.8661475Z ----------------------------------------------------------------------
2022-12-01T10:51:33.8661792Z   test_derive_module_name (__main__.LoggingTest) ... ok (0.265s)
2022-12-01T10:51:33.8662138Z   test_logger_name (__main__.LoggingTest) ... ok (0.002s)
2022-12-01T10:51:33.8662362Z 
2022-12-01T10:51:33.8662607Z ----------------------------------------------------------------------
2022-12-01T10:51:33.8662855Z Ran 2 tests in 0.268s
2022-12-01T10:51:33.8662963Z 
2022-12-01T10:51:33.8663026Z OK
2022-12-01T10:51:33.8663140Z 
2022-12-01T10:51:33.8663258Z Generating XML reports...
2022-12-01T10:51:33.8663736Z Generated XML report: test-reports/python-unittest/distributed.elastic.utils.logging_test/TEST-LoggingTest-20221201105133.xml
2022-12-01T10:51:33.8664002Z 
2022-12-01T10:51:33.8664230Z ##[endgroup]
2022-12-01T10:51:33.8664710Z FINISHED PRINTING LOG FILE of distributed/elastic/utils/logging_test (/var/lib/jenkins/workspace/test/test-reports/distributed-elastic-utils-logging_test_db21yxtq)
2022-12-01T10:51:33.8665029Z 
2022-12-01T10:51:33.8665242Z Running distributed/nn/jit/test_instantiator ... [2022-12-01 10:51:33.866128]
2022-12-01T10:51:33.8666315Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/nn/jit/test_instantiator.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-12-01 10:51:33.866461]
2022-12-01T10:51:35.3377607Z 
2022-12-01T10:51:35.3378229Z Expand the folded group to see the log file of distributed/nn/jit/test_instantiator
2022-12-01T10:51:35.3379350Z ##[group]PRINTING LOG FILE of distributed/nn/jit/test_instantiator (/var/lib/jenkins/workspace/test/test-reports/distributed-nn-jit-test_instantiator_0lte8gxn)
2022-12-01T10:51:35.3380007Z Test results will be stored in test-reports/python-unittest/distributed.nn.jit.test_instantiator
2022-12-01T10:51:35.3380220Z 
2022-12-01T10:51:35.3380294Z Running tests...
2022-12-01T10:51:35.3380782Z ----------------------------------------------------------------------
2022-12-01T10:51:35.3381177Z   test_get_arg_return_types_from_interface (__main__.TestInstantiator) ... ok (0.233s)
2022-12-01T10:51:35.3381525Z   test_instantiate_non_scripted_remote_module_template (__main__.TestInstantiator) ... ok (0.002s)
2022-12-01T10:51:35.3381928Z   test_instantiate_scripted_remote_module_template (__main__.TestInstantiator) ... ok (0.010s)
2022-12-01T10:51:35.3382104Z 
2022-12-01T10:51:35.3382309Z ----------------------------------------------------------------------
2022-12-01T10:51:35.3382554Z Ran 3 tests in 0.246s
2022-12-01T10:51:35.3382669Z 
2022-12-01T10:51:35.3382733Z OK
2022-12-01T10:51:35.3382826Z 
2022-12-01T10:51:35.3382896Z Generating XML reports...
2022-12-01T10:51:35.3383365Z Generated XML report: test-reports/python-unittest/distributed.nn.jit.test_instantiator/TEST-TestInstantiator-20221201105134.xml
2022-12-01T10:51:35.3383624Z 
2022-12-01T10:51:35.3383853Z ##[endgroup]
2022-12-01T10:51:35.3384339Z FINISHED PRINTING LOG FILE of distributed/nn/jit/test_instantiator (/var/lib/jenkins/workspace/test/test-reports/distributed-nn-jit-test_instantiator_0lte8gxn)
2022-12-01T10:51:35.3384647Z 
2022-12-01T10:51:35.3384848Z Running distributed/test_data_parallel ... [2022-12-01 10:51:35.338010]
2022-12-01T10:51:35.3385875Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_data_parallel.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-12-01 10:51:35.338363]
2022-12-01T10:51:36.8478130Z 
2022-12-01T10:51:36.8479020Z Expand the folded group to see the log file of distributed/test_data_parallel
2022-12-01T10:51:36.8480045Z ##[group]PRINTING LOG FILE of distributed/test_data_parallel (/var/lib/jenkins/workspace/test/test-reports/distributed-test_data_parallel_83ve8_0r)
2022-12-01T10:51:36.8480717Z Test results will be stored in test-reports/python-unittest/distributed.test_data_parallel
2022-12-01T10:51:36.8480954Z 
2022-12-01T10:51:36.8481033Z Running tests...
2022-12-01T10:51:36.8481412Z ----------------------------------------------------------------------
2022-12-01T10:51:36.8481911Z   test_autocast (__main__.TestDataParallel) ... skip: multi-GPU not supported (0.001s)
2022-12-01T10:51:36.8482422Z   test_data_parallel (__main__.TestDataParallel) ... skip: multi-GPU not supported (0.001s)
2022-12-01T10:51:36.8482891Z   test_data_parallel_buffers_requiring_grad (__main__.TestDataParallel) ... skip: multi-GPU not supported (0.001s)
2022-12-01T10:51:36.8483255Z   test_data_parallel_complex (__main__.TestDataParallel) ... skip: At least 2 CUDA GPUS needed (0.001s)
2022-12-01T10:51:36.8483706Z   test_data_parallel_device_args (__main__.TestDataParallel) ... skip: multi-GPU not supported (0.001s)
2022-12-01T10:51:36.8484168Z   test_data_parallel_function_deletion (__main__.TestDataParallel) ... skip: multi-GPU not supported (0.001s)
2022-12-01T10:51:36.8484728Z   test_data_parallel_lazy_linear (__main__.TestDataParallel) ... skip: multi-GPU not supported (0.000s)
2022-12-01T10:51:36.8485047Z   test_data_parallel_model_device (__main__.TestDataParallel)
2022-12-01T10:51:36.8485437Z Test device[0] check at forward time. ... skip: multi-GPU not supported (0.003s)
2022-12-01T10:51:36.8485889Z   test_data_parallel_model_no_refcycles (__main__.TestDataParallel) ... skip: multi-GPU not supported (0.001s)
2022-12-01T10:51:36.8486333Z   test_data_parallel_module_zero_inputs (__main__.TestDataParallel) ... skip: multi-GPU not supported (0.001s)
2022-12-01T10:51:36.8486786Z   test_data_parallel_multiple_input (__main__.TestDataParallel) ... skip: multi-GPU not supported (0.002s)
2022-12-01T10:51:36.8487239Z   test_data_parallel_nested_input (__main__.TestDataParallel) ... skip: multi-GPU not supported (0.001s)
2022-12-01T10:51:36.8487689Z   test_data_parallel_nested_output (__main__.TestDataParallel) ... skip: multi-GPU not supported (0.001s)
2022-12-01T10:51:36.8488109Z   test_data_parallel_no_grad (__main__.TestDataParallel) ... skip: multi-GPU not supported (0.001s)
2022-12-01T10:51:36.8488531Z   test_data_parallel_rnn (__main__.TestDataParallel) ... skip: multi-GPU not supported (0.001s)
2022-12-01T10:51:36.8488961Z   test_data_parallel_small_back (__main__.TestDataParallel) ... skip: multi-GPU not supported (0.000s)
2022-12-01T10:51:36.8489393Z   test_data_parallel_sparse (__main__.TestDataParallel) ... skip: multi-GPU not supported (0.001s)
2022-12-01T10:51:36.8489797Z   test_gather_cpu (__main__.TestDataParallel) ... skip: multi-GPU not supported (0.000s)
2022-12-01T10:51:36.8490228Z   test_gather_different_len_dicts (__main__.TestDataParallel) ... skip: multi-GPU not supported (0.000s)
2022-12-01T10:51:36.8490648Z   test_gather_gpu (__main__.TestDataParallel) ... skip: multi-GPU not supported (0.000s)
2022-12-01T10:51:36.8491044Z   test_parallel_apply (__main__.TestDataParallel) ... skip: multi-GPU not supported (0.001s)
2022-12-01T10:51:36.8491475Z   test_parallel_apply_autocast (__main__.TestDataParallel) ... skip: multi-GPU not supported (0.001s)
2022-12-01T10:51:36.8491835Z   test_parallel_apply_passes_exception (__main__.TestDataParallel) ... skip: CUDA unavailable (0.000s)
2022-12-01T10:51:36.8492359Z   test_parameter_list_dict_replica (__main__.TestDataParallel) ... skip: multi-GPU not supported (0.001s)
2022-12-01T10:51:36.8492765Z   test_replicate (__main__.TestDataParallel) ... skip: multi-GPU not supported (0.001s)
2022-12-01T10:51:36.8493243Z   test_replicate_buffers (__main__.TestDataParallel) ... skip: multi-GPU not supported (0.001s)
2022-12-01T10:51:36.8493673Z   test_save_replica_module (__main__.TestDataParallel) ... skip: multi-GPU not supported (0.000s)
2022-12-01T10:51:36.8494144Z   test_scatter_cpu (__main__.TestDataParallel) ... skip: multi-GPU not supported (0.000s)
2022-12-01T10:51:36.8494597Z   test_scatter_gpu (__main__.TestDataParallel) ... skip: multi-GPU not supported (0.000s)
2022-12-01T10:51:36.8495011Z   test_strided_grad_layout (__main__.TestDataParallel) ... skip: multi-GPU not supported (0.003s)
2022-12-01T10:51:36.8495424Z   test_zero_grad (__main__.TestDataParallel) ... skip: multi-GPU not supported (0.001s)
2022-12-01T10:51:36.8495789Z   test_data_parallel_module_cpu_float16 (__main__.TestDataParallelDeviceTypeCPU) ... skip: Only runs on cuda (0.002s)
2022-12-01T10:51:36.8496210Z   test_data_parallel_module_cpu_float32 (__main__.TestDataParallelDeviceTypeCPU) ... skip: Only runs on cuda (0.001s)
2022-12-01T10:51:36.8496619Z   test_data_parallel_module_cpu_float64 (__main__.TestDataParallelDeviceTypeCPU) ... skip: Only runs on cuda (0.001s)
2022-12-01T10:51:36.8497036Z   test_data_parallel_module_kwargs_only_cpu_float16 (__main__.TestDataParallelDeviceTypeCPU) ... skip: Only runs on cuda (0.001s)
2022-12-01T10:51:36.8497453Z   test_data_parallel_module_kwargs_only_cpu_float32 (__main__.TestDataParallelDeviceTypeCPU) ... skip: Only runs on cuda (0.001s)
2022-12-01T10:51:36.8497875Z   test_data_parallel_module_kwargs_only_cpu_float64 (__main__.TestDataParallelDeviceTypeCPU) ... skip: Only runs on cuda (0.001s)
2022-12-01T10:51:36.8498314Z   test_data_parallel_module_kwargs_only_empty_dict_cpu_float16 (__main__.TestDataParallelDeviceTypeCPU) ... skip: Only runs on cuda (0.001s)
2022-12-01T10:51:36.8498766Z   test_data_parallel_module_kwargs_only_empty_dict_cpu_float32 (__main__.TestDataParallelDeviceTypeCPU) ... skip: Only runs on cuda (0.001s)
2022-12-01T10:51:36.8499197Z   test_data_parallel_module_kwargs_only_empty_dict_cpu_float64 (__main__.TestDataParallelDeviceTypeCPU) ... skip: Only runs on cuda (0.001s)
2022-12-01T10:51:36.8499647Z   test_data_parallel_module_kwargs_only_empty_list_cpu_float16 (__main__.TestDataParallelDeviceTypeCPU) ... skip: Only runs on cuda (0.001s)
2022-12-01T10:51:36.8500083Z   test_data_parallel_module_kwargs_only_empty_list_cpu_float32 (__main__.TestDataParallelDeviceTypeCPU) ... skip: Only runs on cuda (0.001s)
2022-12-01T10:51:36.8500722Z   test_data_parallel_module_kwargs_only_empty_list_cpu_float64 (__main__.TestDataParallelDeviceTypeCPU) ... skip: Only runs on cuda (0.001s)
2022-12-01T10:51:36.8501155Z   test_data_parallel_module_kwargs_only_empty_tuple_cpu_float16 (__main__.TestDataParallelDeviceTypeCPU) ... skip: Only runs on cuda (0.001s)
2022-12-01T10:51:36.8501611Z   test_data_parallel_module_kwargs_only_empty_tuple_cpu_float32 (__main__.TestDataParallelDeviceTypeCPU) ... skip: Only runs on cuda (0.001s)
2022-12-01T10:51:36.8502067Z   test_data_parallel_module_kwargs_only_empty_tuple_cpu_float64 (__main__.TestDataParallelDeviceTypeCPU) ... skip: Only runs on cuda (0.001s)
2022-12-01T10:51:36.8502309Z 
2022-12-01T10:51:36.8502515Z ----------------------------------------------------------------------
2022-12-01T10:51:36.8502749Z Ran 46 tests in 0.048s
2022-12-01T10:51:36.8502863Z 
2022-12-01T10:51:36.8502942Z OK (skipped=46)
2022-12-01T10:51:36.8503051Z 
2022-12-01T10:51:36.8503136Z Generating XML reports...
2022-12-01T10:51:36.8503579Z Generated XML report: test-reports/python-unittest/distributed.test_data_parallel/TEST-TestDataParallel-20221201105136.xml
2022-12-01T10:51:36.8504179Z Generated XML report: test-reports/python-unittest/distributed.test_data_parallel/TEST-TestDataParallelDeviceTypeCPU-20221201105136.xml
2022-12-01T10:51:36.8504464Z 
2022-12-01T10:51:36.8504739Z ##[endgroup]
2022-12-01T10:51:36.8505188Z FINISHED PRINTING LOG FILE of distributed/test_data_parallel (/var/lib/jenkins/workspace/test/test-reports/distributed-test_data_parallel_83ve8_0r)
2022-12-01T10:51:36.8505423Z 
2022-12-01T10:51:36.8505669Z Running distributed/_shard/sharded_tensor/ops/test_matrix_ops ... [2022-12-01 10:51:36.848074]
2022-12-01T10:51:36.8506310Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/sharded_tensor/ops/test_matrix_ops.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-12-01 10:51:36.848405]
2022-12-01T10:51:38.0204738Z 
2022-12-01T10:51:38.0206350Z Expand the folded group to see the log file of distributed/_shard/sharded_tensor/ops/test_matrix_ops
2022-12-01T10:51:38.0207533Z ##[group]PRINTING LOG FILE of distributed/_shard/sharded_tensor/ops/test_matrix_ops (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharded_tensor-ops-test_matrix_ops_0ronrero)
2022-12-01T10:51:38.0208603Z Test results will be stored in test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_matrix_ops
2022-12-01T10:51:38.0208975Z 
2022-12-01T10:51:38.0209106Z Running tests...
2022-12-01T10:51:38.0209579Z ----------------------------------------------------------------------
2022-12-01T10:51:38.0210218Z   test_sharded_tensor_contiguous (__main__.TestShardedTensorMatrixOps) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-12-01T10:51:38.0210963Z   test_sharded_tensor_layer_norm (__main__.TestShardedTensorMatrixOps) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-12-01T10:51:38.0211649Z   test_sharded_tensor_layer_norm_error (__main__.TestShardedTensorMatrixOps) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-12-01T10:51:38.0212411Z   test_sharded_tensor_masked_fill (__main__.TestShardedTensorMatrixOps) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2022-12-01T10:51:38.0213107Z   test_sharded_tensor_masked_fill_error (__main__.TestShardedTensorMatrixOps) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-12-01T10:51:38.0213748Z   test_sharded_tensor_softmax (__main__.TestShardedTensorMatrixOps) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-12-01T10:51:38.0214332Z   test_sharded_tensor_transpose (__main__.TestShardedTensorMatrixOps) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-12-01T10:51:38.0214923Z   test_sharded_tensor_transpose_error (__main__.TestShardedTensorMatrixOps) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2022-12-01T10:51:38.0215505Z   test_sharded_tensor_type_as (__main__.TestShardedTensorMatrixOps) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-12-01T10:51:38.0216081Z   test_sharded_tensor_view (__main__.TestShardedTensorMatrixOps) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-12-01T10:51:38.0216668Z   test_sharded_tensor_view_error (__main__.TestShardedTensorMatrixOps) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-12-01T10:51:38.0216984Z 
2022-12-01T10:51:38.0217327Z ----------------------------------------------------------------------
2022-12-01T10:51:38.0217653Z Ran 11 tests in 0.008s
2022-12-01T10:51:38.0217815Z 
2022-12-01T10:51:38.0217917Z OK (skipped=11)
2022-12-01T10:51:38.0218071Z 
2022-12-01T10:51:38.0218189Z Generating XML reports...
2022-12-01T10:51:38.0218937Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_matrix_ops/TEST-TestShardedTensorMatrixOps-20221201105137.xml
2022-12-01T10:51:38.0219408Z 
2022-12-01T10:51:38.0219787Z ##[endgroup]
2022-12-01T10:51:38.0220802Z FINISHED PRINTING LOG FILE of distributed/_shard/sharded_tensor/ops/test_matrix_ops (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharded_tensor-ops-test_matrix_ops_0ronrero)
2022-12-01T10:51:38.0221235Z 
2022-12-01T10:51:38.0221588Z Running distributed/_shard/sharded_tensor/ops/test_linear ... [2022-12-01 10:51:38.020755]
2022-12-01T10:51:38.0222378Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/sharded_tensor/ops/test_linear.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-12-01 10:51:38.021155]
2022-12-01T10:51:39.1889628Z 
2022-12-01T10:51:39.1890188Z Expand the folded group to see the log file of distributed/_shard/sharded_tensor/ops/test_linear
2022-12-01T10:51:39.1891422Z ##[group]PRINTING LOG FILE of distributed/_shard/sharded_tensor/ops/test_linear (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharded_tensor-ops-test_linear_afbqf_oz)
2022-12-01T10:51:39.1892252Z Test results will be stored in test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_linear
2022-12-01T10:51:39.1892477Z 
2022-12-01T10:51:39.1892554Z Running tests...
2022-12-01T10:51:39.1892851Z ----------------------------------------------------------------------
2022-12-01T10:51:39.1893272Z   test_sharded_linear_colwise (__main__.TestShardedTensorOpsLinear) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-12-01T10:51:39.1893776Z   test_sharded_linear_errors (__main__.TestShardedTensorOpsLinear) ... skip: c10d was not compiled with the NCCL backend (0.003s)
2022-12-01T10:51:39.1894226Z   test_sharded_linear_rowwise (__main__.TestShardedTensorOpsLinear) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-12-01T10:51:39.1894450Z 
2022-12-01T10:51:39.1894637Z ----------------------------------------------------------------------
2022-12-01T10:51:39.1894884Z Ran 3 tests in 0.006s
2022-12-01T10:51:39.1895000Z 
2022-12-01T10:51:39.1895079Z OK (skipped=3)
2022-12-01T10:51:39.1895190Z 
2022-12-01T10:51:39.1895275Z Generating XML reports...
2022-12-01T10:51:39.1895780Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_linear/TEST-TestShardedTensorOpsLinear-20221201105138.xml
2022-12-01T10:51:39.1896075Z 
2022-12-01T10:51:39.1896382Z ##[endgroup]
2022-12-01T10:51:39.1897038Z FINISHED PRINTING LOG FILE of distributed/_shard/sharded_tensor/ops/test_linear (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharded_tensor-ops-test_linear_afbqf_oz)
2022-12-01T10:51:39.1897555Z 
2022-12-01T10:51:39.1897898Z Running distributed/_shard/sharded_tensor/ops/test_elementwise_ops ... [2022-12-01 10:51:39.189232]
2022-12-01T10:51:39.1898486Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/sharded_tensor/ops/test_elementwise_ops.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-12-01 10:51:39.189566]
2022-12-01T10:51:40.3492300Z 
2022-12-01T10:51:40.3492928Z Expand the folded group to see the log file of distributed/_shard/sharded_tensor/ops/test_elementwise_ops
2022-12-01T10:51:40.3494236Z ##[group]PRINTING LOG FILE of distributed/_shard/sharded_tensor/ops/test_elementwise_ops (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharded_tensor-ops-test_elementwise_ops_xoe2j4oe)
2022-12-01T10:51:40.3495495Z Test results will be stored in test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_elementwise_ops
2022-12-01T10:51:40.3495817Z 
2022-12-01T10:51:40.3495930Z Running tests...
2022-12-01T10:51:40.3496384Z ----------------------------------------------------------------------
2022-12-01T10:51:40.3497095Z   test_sharded_dropout (__main__.TestShardedTensorElementWiseOps) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-12-01T10:51:40.3497538Z   test_sharded_gelu (__main__.TestShardedTensorElementWiseOps) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-12-01T10:51:40.3497971Z   test_sharded_relu (__main__.TestShardedTensorElementWiseOps) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-12-01T10:51:40.3498420Z   test_sharded_tensor_nan_to_num (__main__.TestShardedTensorElementWiseOps) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-12-01T10:51:40.3498660Z 
2022-12-01T10:51:40.3498911Z ----------------------------------------------------------------------
2022-12-01T10:51:40.3499176Z Ran 4 tests in 0.003s
2022-12-01T10:51:40.3499369Z 
2022-12-01T10:51:40.3499488Z OK (skipped=4)
2022-12-01T10:51:40.3499678Z 
2022-12-01T10:51:40.3499837Z Generating XML reports...
2022-12-01T10:51:40.3500675Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_elementwise_ops/TEST-TestShardedTensorElementWiseOps-20221201105140.xml
2022-12-01T10:51:40.3500985Z 
2022-12-01T10:51:40.3501233Z ##[endgroup]
2022-12-01T10:51:40.3501955Z FINISHED PRINTING LOG FILE of distributed/_shard/sharded_tensor/ops/test_elementwise_ops (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharded_tensor-ops-test_elementwise_ops_xoe2j4oe)
2022-12-01T10:51:40.3502333Z 
2022-12-01T10:51:40.3502541Z Running distributed/test_c10d_spawn_nccl ... [2022-12-01 10:51:40.349456]
2022-12-01T10:51:40.3503072Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_c10d_spawn_nccl.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-12-01 10:51:40.349785]
2022-12-01T10:52:00.5707200Z 
2022-12-01T10:52:00.5708258Z Expand the folded group to see the log file of distributed/test_c10d_spawn_nccl
2022-12-01T10:52:00.5709047Z ##[group]PRINTING LOG FILE of distributed/test_c10d_spawn_nccl (/var/lib/jenkins/workspace/test/test-reports/distributed-test_c10d_spawn_nccl_m_rdspc4)
2022-12-01T10:52:00.5709478Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgj5waoth
2022-12-01T10:52:00.5709988Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgj5waoth/_remote_module_non_scriptable.py
2022-12-01T10:52:00.5710772Z <unittest.suite.TestSuite tests=[<__main__.ProcessGroupShareTensorTest testMethod=test_shared_allgather_nccl>, <__main__.ProcessGroupShareTensorTest testMethod=test_shared_allreduce_nccl>, <__main__.ProcessGroupShareTensorTest testMethod=test_shared_broadcast_nccl>, <__main__.ProcessGroupShareTensorTest testMethod=test_shared_reduce_nccl>]>
2022-12-01T10:52:00.5711573Z test_shared_allgather_nccl (__main__.ProcessGroupShareTensorTest)
2022-12-01T10:52:00.5711904Z test_shared_allreduce_nccl (__main__.ProcessGroupShareTensorTest)
2022-12-01T10:52:00.5712258Z test_shared_broadcast_nccl (__main__.ProcessGroupShareTensorTest)
2022-12-01T10:52:00.5712603Z test_shared_reduce_nccl (__main__.ProcessGroupShareTensorTest)
2022-12-01T10:52:00.5713020Z <unittest.suite.TestSuite tests=[]>
2022-12-01T10:52:00.5713346Z <unittest.suite.TestSuite tests=[]>
2022-12-01T10:52:00.5714360Z <unittest.suite.TestSuite tests=[<__main__.TestDistributedNNFunctionsNccl testMethod=test_all_gather>, <__main__.TestDistributedNNFunctionsNccl testMethod=test_all_gather_base>, <__main__.TestDistributedNNFunctionsNccl testMethod=test_all_to_all>, <__main__.TestDistributedNNFunctionsNccl testMethod=test_all_to_all_single>, <__main__.TestDistributedNNFunctionsNccl testMethod=test_allreduce>, <__main__.TestDistributedNNFunctionsNccl testMethod=test_broadcast>, <__main__.TestDistributedNNFunctionsNccl testMethod=test_reduce>, <__main__.TestDistributedNNFunctionsNccl testMethod=test_reduce_scatter>, <__main__.TestDistributedNNFunctionsNccl testMethod=test_reduce_scatter_non_contiguous>]>
2022-12-01T10:52:00.5715363Z test_all_gather (__main__.TestDistributedNNFunctionsNccl)
2022-12-01T10:52:00.5715671Z test_all_gather_base (__main__.TestDistributedNNFunctionsNccl)
2022-12-01T10:52:00.5715973Z test_all_to_all (__main__.TestDistributedNNFunctionsNccl)
2022-12-01T10:52:00.5716320Z test_all_to_all_single (__main__.TestDistributedNNFunctionsNccl)
2022-12-01T10:52:00.5716695Z test_allreduce (__main__.TestDistributedNNFunctionsNccl)
2022-12-01T10:52:00.5717056Z test_broadcast (__main__.TestDistributedNNFunctionsNccl)
2022-12-01T10:52:00.5717335Z test_reduce (__main__.TestDistributedNNFunctionsNccl)
2022-12-01T10:52:00.5717635Z test_reduce_scatter (__main__.TestDistributedNNFunctionsNccl)
2022-12-01T10:52:00.5717958Z test_reduce_scatter_non_contiguous (__main__.TestDistributedNNFunctionsNccl)
2022-12-01T10:52:00.5718518Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:52:00.5718858Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:52:00.5719302Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:52:00.5719655Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:52:00.5720175Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk6wbafjj
2022-12-01T10:52:00.5720590Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk6wbafjj/_remote_module_non_scriptable.py
2022-12-01T10:52:00.5721127Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_nccl
2022-12-01T10:52:00.5721329Z 
2022-12-01T10:52:00.5721403Z Running tests...
2022-12-01T10:52:00.5721700Z ----------------------------------------------------------------------
2022-12-01T10:52:00.5722069Z   test_shared_allgather_nccl (__main__.ProcessGroupShareTensorTest) ... skip: At least 2 CUDA GPUS needed (0.000s)
2022-12-01T10:52:00.5722291Z 
2022-12-01T10:52:00.5722486Z ----------------------------------------------------------------------
2022-12-01T10:52:00.5722714Z Ran 1 test in 0.001s
2022-12-01T10:52:00.5722828Z 
2022-12-01T10:52:00.5722901Z OK (skipped=1)
2022-12-01T10:52:00.5723013Z 
2022-12-01T10:52:00.5723099Z Generating XML reports...
2022-12-01T10:52:00.5723590Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-ProcessGroupShareTensorTest-20221201105142.xml
2022-12-01T10:52:00.5724139Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:52:00.5724473Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:52:00.5724906Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:52:00.5725241Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:52:00.5725594Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptertvr21
2022-12-01T10:52:00.5726000Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptertvr21/_remote_module_non_scriptable.py
2022-12-01T10:52:00.5726457Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_nccl
2022-12-01T10:52:00.5726648Z 
2022-12-01T10:52:00.5726723Z Running tests...
2022-12-01T10:52:00.5727026Z ----------------------------------------------------------------------
2022-12-01T10:52:00.5727395Z   test_shared_allreduce_nccl (__main__.ProcessGroupShareTensorTest) ... skip: At least 2 CUDA GPUS needed (0.000s)
2022-12-01T10:52:00.5727614Z 
2022-12-01T10:52:00.5727806Z ----------------------------------------------------------------------
2022-12-01T10:52:00.5728033Z Ran 1 test in 0.001s
2022-12-01T10:52:00.5728147Z 
2022-12-01T10:52:00.5728218Z OK (skipped=1)
2022-12-01T10:52:00.5728325Z 
2022-12-01T10:52:00.5728409Z Generating XML reports...
2022-12-01T10:52:00.5728871Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-ProcessGroupShareTensorTest-20221201105144.xml
2022-12-01T10:52:00.5729425Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:52:00.5729756Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:52:00.5730190Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:52:00.5730525Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:52:00.5730873Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf72s_6sd
2022-12-01T10:52:00.5731273Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf72s_6sd/_remote_module_non_scriptable.py
2022-12-01T10:52:00.5731712Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_nccl
2022-12-01T10:52:00.5731991Z 
2022-12-01T10:52:00.5732069Z Running tests...
2022-12-01T10:52:00.5732373Z ----------------------------------------------------------------------
2022-12-01T10:52:00.5732736Z   test_shared_broadcast_nccl (__main__.ProcessGroupShareTensorTest) ... skip: At least 2 CUDA GPUS needed (0.000s)
2022-12-01T10:52:00.5732941Z 
2022-12-01T10:52:00.5733132Z ----------------------------------------------------------------------
2022-12-01T10:52:00.5733428Z Ran 1 test in 0.001s
2022-12-01T10:52:00.5733550Z 
2022-12-01T10:52:00.5733660Z OK (skipped=1)
2022-12-01T10:52:00.5733771Z 
2022-12-01T10:52:00.5733841Z Generating XML reports...
2022-12-01T10:52:00.5734324Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-ProcessGroupShareTensorTest-20221201105145.xml
2022-12-01T10:52:00.5734876Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:52:00.5735214Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:52:00.5735629Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:52:00.5735979Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:52:00.5736329Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwj7tfhmi
2022-12-01T10:52:00.5736725Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwj7tfhmi/_remote_module_non_scriptable.py
2022-12-01T10:52:00.5737179Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_nccl
2022-12-01T10:52:00.5737377Z 
2022-12-01T10:52:00.5737449Z Running tests...
2022-12-01T10:52:00.5737753Z ----------------------------------------------------------------------
2022-12-01T10:52:00.5738097Z   test_shared_reduce_nccl (__main__.ProcessGroupShareTensorTest) ... skip: At least 2 CUDA GPUS needed (0.000s)
2022-12-01T10:52:00.5738316Z 
2022-12-01T10:52:00.5738510Z ----------------------------------------------------------------------
2022-12-01T10:52:00.5738746Z Ran 1 test in 0.001s
2022-12-01T10:52:00.5738861Z 
2022-12-01T10:52:00.5738931Z OK (skipped=1)
2022-12-01T10:52:00.5739025Z 
2022-12-01T10:52:00.5739111Z Generating XML reports...
2022-12-01T10:52:00.5739582Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-ProcessGroupShareTensorTest-20221201105147.xml
2022-12-01T10:52:00.5740139Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:52:00.5740655Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:52:00.5741096Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:52:00.5741450Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:52:00.5741807Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi74fquzd
2022-12-01T10:52:00.5742198Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi74fquzd/_remote_module_non_scriptable.py
2022-12-01T10:52:00.5742658Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_nccl
2022-12-01T10:52:00.5742857Z 
2022-12-01T10:52:00.5742931Z Running tests...
2022-12-01T10:52:00.5743219Z ----------------------------------------------------------------------
2022-12-01T10:52:00.5743647Z   test_all_gather (__main__.TestDistributedNNFunctionsNccl) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2022-12-01T10:52:00.5743873Z 
2022-12-01T10:52:00.5744067Z ----------------------------------------------------------------------
2022-12-01T10:52:00.5744305Z Ran 1 test in 0.000s
2022-12-01T10:52:00.5744418Z 
2022-12-01T10:52:00.5744477Z OK (skipped=1)
2022-12-01T10:52:00.5744585Z 
2022-12-01T10:52:00.5744670Z Generating XML reports...
2022-12-01T10:52:00.5745159Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20221201105148.xml
2022-12-01T10:52:00.5745716Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:52:00.5746039Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:52:00.5746533Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:52:00.5746886Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:52:00.5747265Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuj1qhk94
2022-12-01T10:52:00.5747670Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuj1qhk94/_remote_module_non_scriptable.py
2022-12-01T10:52:00.5748126Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_nccl
2022-12-01T10:52:00.5748324Z 
2022-12-01T10:52:00.5748397Z Running tests...
2022-12-01T10:52:00.5748685Z ----------------------------------------------------------------------
2022-12-01T10:52:00.5749061Z   test_all_gather_base (__main__.TestDistributedNNFunctionsNccl) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-12-01T10:52:00.5749289Z 
2022-12-01T10:52:00.5749483Z ----------------------------------------------------------------------
2022-12-01T10:52:00.5749708Z Ran 1 test in 0.001s
2022-12-01T10:52:00.5749823Z 
2022-12-01T10:52:00.5749896Z OK (skipped=1)
2022-12-01T10:52:00.5750007Z 
2022-12-01T10:52:00.5750092Z Generating XML reports...
2022-12-01T10:52:00.5750575Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20221201105149.xml
2022-12-01T10:52:00.5751118Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:52:00.5751451Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:52:00.5751877Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:52:00.5752225Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:52:00.5752558Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpod6uh9ra
2022-12-01T10:52:00.5752964Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpod6uh9ra/_remote_module_non_scriptable.py
2022-12-01T10:52:00.5753413Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_nccl
2022-12-01T10:52:00.5753611Z 
2022-12-01T10:52:00.5753671Z Running tests...
2022-12-01T10:52:00.5753979Z ----------------------------------------------------------------------
2022-12-01T10:52:00.5754349Z   test_all_to_all (__main__.TestDistributedNNFunctionsNccl) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2022-12-01T10:52:00.5754571Z 
2022-12-01T10:52:00.5754765Z ----------------------------------------------------------------------
2022-12-01T10:52:00.5754986Z Ran 1 test in 0.000s
2022-12-01T10:52:00.5755099Z 
2022-12-01T10:52:00.5755172Z OK (skipped=1)
2022-12-01T10:52:00.5755282Z 
2022-12-01T10:52:00.5755365Z Generating XML reports...
2022-12-01T10:52:00.5755834Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20221201105151.xml
2022-12-01T10:52:00.5756395Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:52:00.5756729Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:52:00.5757156Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:52:00.5757485Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:52:00.5757853Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppsmlcw1z
2022-12-01T10:52:00.5758256Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppsmlcw1z/_remote_module_non_scriptable.py
2022-12-01T10:52:00.5758695Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_nccl
2022-12-01T10:52:00.5758893Z 
2022-12-01T10:52:00.5758964Z Running tests...
2022-12-01T10:52:00.5759268Z ----------------------------------------------------------------------
2022-12-01T10:52:00.5759691Z   test_all_to_all_single (__main__.TestDistributedNNFunctionsNccl) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2022-12-01T10:52:00.5759959Z 
2022-12-01T10:52:00.5760140Z ----------------------------------------------------------------------
2022-12-01T10:52:00.5760377Z Ran 1 test in 0.000s
2022-12-01T10:52:00.5760490Z 
2022-12-01T10:52:00.5760564Z OK (skipped=1)
2022-12-01T10:52:00.5760672Z 
2022-12-01T10:52:00.5760758Z Generating XML reports...
2022-12-01T10:52:00.5761230Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20221201105152.xml
2022-12-01T10:52:00.5761784Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:52:00.5762126Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:52:00.5762544Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:52:00.5762896Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:52:00.5763243Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp77l6udls
2022-12-01T10:52:00.5763646Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp77l6udls/_remote_module_non_scriptable.py
2022-12-01T10:52:00.5764087Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_nccl
2022-12-01T10:52:00.5764280Z 
2022-12-01T10:52:00.5764352Z Running tests...
2022-12-01T10:52:00.5764650Z ----------------------------------------------------------------------
2022-12-01T10:52:00.5765005Z   test_allreduce (__main__.TestDistributedNNFunctionsNccl) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2022-12-01T10:52:00.5765231Z 
2022-12-01T10:52:00.5765425Z ----------------------------------------------------------------------
2022-12-01T10:52:00.5765659Z Ran 1 test in 0.000s
2022-12-01T10:52:00.5765772Z 
2022-12-01T10:52:00.5765845Z OK (skipped=1)
2022-12-01T10:52:00.5765953Z 
2022-12-01T10:52:00.5766029Z Generating XML reports...
2022-12-01T10:52:00.5766510Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20221201105154.xml
2022-12-01T10:52:00.5767066Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:52:00.5767397Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:52:00.5767812Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:52:00.5768158Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:52:00.5768503Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5tu2gljh
2022-12-01T10:52:00.5768892Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5tu2gljh/_remote_module_non_scriptable.py
2022-12-01T10:52:00.5769342Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_nccl
2022-12-01T10:52:00.5769543Z 
2022-12-01T10:52:00.5769615Z Running tests...
2022-12-01T10:52:00.5769919Z ----------------------------------------------------------------------
2022-12-01T10:52:00.5770277Z   test_broadcast (__main__.TestDistributedNNFunctionsNccl) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2022-12-01T10:52:00.5770502Z 
2022-12-01T10:52:00.5770696Z ----------------------------------------------------------------------
2022-12-01T10:52:00.5770933Z Ran 1 test in 0.000s
2022-12-01T10:52:00.5771048Z 
2022-12-01T10:52:00.5771106Z OK (skipped=1)
2022-12-01T10:52:00.5771214Z 
2022-12-01T10:52:00.5771299Z Generating XML reports...
2022-12-01T10:52:00.5771781Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20221201105155.xml
2022-12-01T10:52:00.5772510Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:52:00.5772864Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:52:00.5773292Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:52:00.5773644Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:52:00.5773990Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgzroj2da
2022-12-01T10:52:00.5774388Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgzroj2da/_remote_module_non_scriptable.py
2022-12-01T10:52:00.5774843Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_nccl
2022-12-01T10:52:00.5775039Z 
2022-12-01T10:52:00.5775114Z Running tests...
2022-12-01T10:52:00.5775403Z ----------------------------------------------------------------------
2022-12-01T10:52:00.5775775Z   test_reduce (__main__.TestDistributedNNFunctionsNccl) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2022-12-01T10:52:00.5775998Z 
2022-12-01T10:52:00.5776193Z ----------------------------------------------------------------------
2022-12-01T10:52:00.5776433Z Ran 1 test in 0.000s
2022-12-01T10:52:00.5776532Z 
2022-12-01T10:52:00.5776607Z OK (skipped=1)
2022-12-01T10:52:00.5776717Z 
2022-12-01T10:52:00.5776801Z Generating XML reports...
2022-12-01T10:52:00.5777282Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20221201105157.xml
2022-12-01T10:52:00.5777830Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:52:00.5778160Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:52:00.5778592Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:52:00.5778998Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:52:00.5779330Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5ef1f2c1
2022-12-01T10:52:00.5779728Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5ef1f2c1/_remote_module_non_scriptable.py
2022-12-01T10:52:00.5780179Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_nccl
2022-12-01T10:52:00.5780508Z 
2022-12-01T10:52:00.5780583Z Running tests...
2022-12-01T10:52:00.5780875Z ----------------------------------------------------------------------
2022-12-01T10:52:00.5781256Z   test_reduce_scatter (__main__.TestDistributedNNFunctionsNccl) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-12-01T10:52:00.5781488Z 
2022-12-01T10:52:00.5781683Z ----------------------------------------------------------------------
2022-12-01T10:52:00.5781908Z Ran 1 test in 0.001s
2022-12-01T10:52:00.5782021Z 
2022-12-01T10:52:00.5782094Z OK (skipped=1)
2022-12-01T10:52:00.5782205Z 
2022-12-01T10:52:00.5782291Z Generating XML reports...
2022-12-01T10:52:00.5782782Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20221201105158.xml
2022-12-01T10:52:00.5783329Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:52:00.5783662Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:52:00.5784094Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:52:00.5784427Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:52:00.5784773Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzlylhs42
2022-12-01T10:52:00.5785175Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzlylhs42/_remote_module_non_scriptable.py
2022-12-01T10:52:00.5785700Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_nccl
2022-12-01T10:52:00.5785927Z 
2022-12-01T10:52:00.5786002Z Running tests...
2022-12-01T10:52:00.5786306Z ----------------------------------------------------------------------
2022-12-01T10:52:00.5786702Z   test_reduce_scatter_non_contiguous (__main__.TestDistributedNNFunctionsNccl) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-12-01T10:52:00.5786943Z 
2022-12-01T10:52:00.5787136Z ----------------------------------------------------------------------
2022-12-01T10:52:00.5787362Z Ran 1 test in 0.001s
2022-12-01T10:52:00.5787480Z 
2022-12-01T10:52:00.5787554Z OK (skipped=1)
2022-12-01T10:52:00.5787662Z 
2022-12-01T10:52:00.5787748Z Generating XML reports...
2022-12-01T10:52:00.5788215Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20221201105200.xml
2022-12-01T10:52:00.5788497Z 
2022-12-01T10:52:00.5788833Z ##[endgroup]
2022-12-01T10:52:00.5789280Z FINISHED PRINTING LOG FILE of distributed/test_c10d_spawn_nccl (/var/lib/jenkins/workspace/test/test-reports/distributed-test_c10d_spawn_nccl_m_rdspc4)
2022-12-01T10:52:00.5789536Z 
2022-12-01T10:52:00.5789765Z Running distributed/_shard/sharded_tensor/ops/test_chunk ... [2022-12-01 10:52:00.571188]
2022-12-01T10:52:00.5801488Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/sharded_tensor/ops/test_chunk.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-12-01 10:52:00.571568]
2022-12-01T10:52:01.7545328Z 
2022-12-01T10:52:01.7545971Z Expand the folded group to see the log file of distributed/_shard/sharded_tensor/ops/test_chunk
2022-12-01T10:52:01.7546982Z ##[group]PRINTING LOG FILE of distributed/_shard/sharded_tensor/ops/test_chunk (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharded_tensor-ops-test_chunk_8jn83ves)
2022-12-01T10:52:01.7547640Z Test results will be stored in test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_chunk
2022-12-01T10:52:01.7547861Z 
2022-12-01T10:52:01.7547946Z Running tests...
2022-12-01T10:52:01.7548295Z ----------------------------------------------------------------------
2022-12-01T10:52:01.7548692Z   test_sharded_chunk (__main__.TestShardedTensorChunkOps) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-12-01T10:52:01.7549230Z   test_sharded_chunk_error (__main__.TestShardedTensorChunkOps) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-12-01T10:52:01.7549494Z 
2022-12-01T10:52:01.7549738Z ----------------------------------------------------------------------
2022-12-01T10:52:01.7549985Z Ran 2 tests in 0.001s
2022-12-01T10:52:01.7550101Z 
2022-12-01T10:52:01.7550161Z OK (skipped=2)
2022-12-01T10:52:01.7550270Z 
2022-12-01T10:52:01.7550354Z Generating XML reports...
2022-12-01T10:52:01.7550873Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_chunk/TEST-TestShardedTensorChunkOps-20221201105201.xml
2022-12-01T10:52:01.7551172Z 
2022-12-01T10:52:01.7551392Z ##[endgroup]
2022-12-01T10:52:01.7552108Z FINISHED PRINTING LOG FILE of distributed/_shard/sharded_tensor/ops/test_chunk (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharded_tensor-ops-test_chunk_8jn83ves)
2022-12-01T10:52:01.7552467Z 
2022-12-01T10:52:01.7552721Z Running distributed/_shard/sharded_tensor/test_megatron_prototype ... [2022-12-01 10:52:01.754770]
2022-12-01T10:52:01.7553309Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/sharded_tensor/test_megatron_prototype.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-12-01 10:52:01.755104]
2022-12-01T10:52:02.9224177Z 
2022-12-01T10:52:02.9224987Z Expand the folded group to see the log file of distributed/_shard/sharded_tensor/test_megatron_prototype
2022-12-01T10:52:02.9226208Z ##[group]PRINTING LOG FILE of distributed/_shard/sharded_tensor/test_megatron_prototype (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharded_tensor-test_megatron_prototype_rs4a1r6x)
2022-12-01T10:52:02.9226909Z Test results will be stored in test-reports/python-unittest/distributed._shard.sharded_tensor.test_megatron_prototype
2022-12-01T10:52:02.9227222Z 
2022-12-01T10:52:02.9227354Z Running tests...
2022-12-01T10:52:02.9227718Z ----------------------------------------------------------------------
2022-12-01T10:52:02.9228281Z   test_megatron_two_layer_prototype (__main__.TestShardedTensorMegatronLinear) ... skip: c10d was not compiled with the NCCL backend (0.002s)
2022-12-01T10:52:02.9228680Z 
2022-12-01T10:52:02.9229058Z ----------------------------------------------------------------------
2022-12-01T10:52:02.9229369Z Ran 1 test in 0.002s
2022-12-01T10:52:02.9229484Z 
2022-12-01T10:52:02.9229559Z OK (skipped=1)
2022-12-01T10:52:02.9229669Z 
2022-12-01T10:52:02.9229758Z Generating XML reports...
2022-12-01T10:52:02.9230305Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.test_megatron_prototype/TEST-TestShardedTensorMegatronLinear-20221201105202.xml
2022-12-01T10:52:02.9230762Z 
2022-12-01T10:52:02.9231032Z ##[endgroup]
2022-12-01T10:52:02.9231980Z FINISHED PRINTING LOG FILE of distributed/_shard/sharded_tensor/test_megatron_prototype (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharded_tensor-test_megatron_prototype_rs4a1r6x)
2022-12-01T10:52:02.9232341Z 
2022-12-01T10:52:02.9232585Z Running distributed/_shard/sharded_tensor/ops/test_binary_cmp ... [2022-12-01 10:52:02.922670]
2022-12-01T10:52:02.9233139Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/sharded_tensor/ops/test_binary_cmp.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-12-01 10:52:02.922995]
2022-12-01T10:52:04.0756791Z 
2022-12-01T10:52:04.0757458Z Expand the folded group to see the log file of distributed/_shard/sharded_tensor/ops/test_binary_cmp
2022-12-01T10:52:04.0758372Z ##[group]PRINTING LOG FILE of distributed/_shard/sharded_tensor/ops/test_binary_cmp (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharded_tensor-ops-test_binary_cmp_bpaxhwpf)
2022-12-01T10:52:04.0759365Z Test results will be stored in test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_binary_cmp
2022-12-01T10:52:04.0759815Z 
2022-12-01T10:52:04.0759924Z Running tests...
2022-12-01T10:52:04.0760283Z ----------------------------------------------------------------------
2022-12-01T10:52:04.0760637Z   test_torch_allclose (__main__.TestShardedTensorBinaryOps)
2022-12-01T10:52:04.0761127Z Test torch.allclose(ShardedTensor, ShardedTensor) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2022-12-01T10:52:04.0761556Z   test_torch_allclose_tensor_specs (__main__.TestShardedTensorBinaryOps) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2022-12-01T10:52:04.0764706Z   test_torch_equal (__main__.TestShardedTensorBinaryOps)
2022-12-01T10:52:04.0765261Z Test torch.equal(ShardedTensor, ShardedTensor) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2022-12-01T10:52:04.0765904Z   test_torch_equal_tensor_specs (__main__.TestShardedTensorBinaryOps) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2022-12-01T10:52:04.0766142Z 
2022-12-01T10:52:04.0766383Z ----------------------------------------------------------------------
2022-12-01T10:52:04.0766631Z Ran 4 tests in 0.002s
2022-12-01T10:52:04.0766733Z 
2022-12-01T10:52:04.0766807Z OK (skipped=4)
2022-12-01T10:52:04.0766916Z 
2022-12-01T10:52:04.0767002Z Generating XML reports...
2022-12-01T10:52:04.0767531Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_binary_cmp/TEST-TestShardedTensorBinaryOps-20221201105203.xml
2022-12-01T10:52:04.0768025Z 
2022-12-01T10:52:04.0768847Z ##[endgroup]
2022-12-01T10:52:04.0769504Z FINISHED PRINTING LOG FILE of distributed/_shard/sharded_tensor/ops/test_binary_cmp (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharded_tensor-ops-test_binary_cmp_bpaxhwpf)
2022-12-01T10:52:04.0769818Z 
2022-12-01T10:52:04.0770226Z Running distributed/_shard/sharded_tensor/ops/test_math_ops ... [2022-12-01 10:52:04.075944]
2022-12-01T10:52:04.0770861Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/sharded_tensor/ops/test_math_ops.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-12-01 10:52:04.076289]
2022-12-01T10:52:05.2357689Z 
2022-12-01T10:52:05.2359201Z Expand the folded group to see the log file of distributed/_shard/sharded_tensor/ops/test_math_ops
2022-12-01T10:52:05.2360054Z ##[group]PRINTING LOG FILE of distributed/_shard/sharded_tensor/ops/test_math_ops (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharded_tensor-ops-test_math_ops__b1dk1j6)
2022-12-01T10:52:05.2360357Z 
2022-12-01T10:52:05.2360575Z ##[endgroup]
2022-12-01T10:52:05.2361264Z FINISHED PRINTING LOG FILE of distributed/_shard/sharded_tensor/ops/test_math_ops (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharded_tensor-ops-test_math_ops__b1dk1j6)
2022-12-01T10:52:05.2361702Z 
2022-12-01T10:52:05.2362430Z Running distributed/_shard/test_sharder ... [2022-12-01 10:52:05.236023]
2022-12-01T10:52:05.2365458Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/test_sharder.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-12-01 10:52:05.236341]
2022-12-01T10:52:06.4061008Z 
2022-12-01T10:52:06.4061549Z Expand the folded group to see the log file of distributed/_shard/test_sharder
2022-12-01T10:52:06.4062619Z ##[group]PRINTING LOG FILE of distributed/_shard/test_sharder (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-test_sharder_ozkyyn4i)
2022-12-01T10:52:06.4063092Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyun31euq
2022-12-01T10:52:06.4063504Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyun31euq/_remote_module_non_scriptable.py
2022-12-01T10:52:06.4063727Z 
2022-12-01T10:52:06.4063930Z ##[endgroup]
2022-12-01T10:52:06.4064481Z FINISHED PRINTING LOG FILE of distributed/_shard/test_sharder (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-test_sharder_ozkyyn4i)
2022-12-01T10:52:06.4064735Z 
2022-12-01T10:52:06.4065525Z Running distributed/elastic/timer/api_test ... [2022-12-01 10:52:06.406372]
2022-12-01T10:52:06.4068517Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/elastic/timer/api_test.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-12-01 10:52:06.406688]
2022-12-01T10:52:07.3647430Z 
2022-12-01T10:52:07.3647911Z Expand the folded group to see the log file of distributed/elastic/timer/api_test
2022-12-01T10:52:07.3649023Z ##[group]PRINTING LOG FILE of distributed/elastic/timer/api_test (/var/lib/jenkins/workspace/test/test-reports/distributed-elastic-timer-api_test_c3840guc)
2022-12-01T10:52:07.3649309Z 
2022-12-01T10:52:07.3649530Z ##[endgroup]
2022-12-01T10:52:07.3650088Z FINISHED PRINTING LOG FILE of distributed/elastic/timer/api_test (/var/lib/jenkins/workspace/test/test-reports/distributed-elastic-timer-api_test_c3840guc)
2022-12-01T10:52:07.3650376Z 
2022-12-01T10:52:07.3652094Z Running distributed/pipeline/sync/skip/test_api ... [2022-12-01 10:52:07.365017]
2022-12-01T10:52:07.3655320Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/skip/test_api.py', '-v'] ... [2022-12-01 10:52:07.365354]
2022-12-01T10:52:09.1272170Z 
2022-12-01T10:52:09.1272667Z Expand the folded group to see the log file of distributed/pipeline/sync/skip/test_api
2022-12-01T10:52:09.1273739Z ##[group]PRINTING LOG FILE of distributed/pipeline/sync/skip/test_api (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-skip-test_api_n_smlu88)
2022-12-01T10:52:09.1274490Z ============================= test session starts ==============================
2022-12-01T10:52:09.1275299Z platform linux -- Python 3.7.13, pytest-7.1.3, pluggy-1.0.0 -- /opt/conda/bin/python
2022-12-01T10:52:09.1275638Z cachedir: .pytest_cache
2022-12-01T10:52:09.1276339Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples')
2022-12-01T10:52:09.1276940Z torch: 1.13.0a0+gitc13d400
2022-12-01T10:52:09.1277389Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini
2022-12-01T10:52:09.1277876Z plugins: hypothesis-5.35.1, forked-1.4.0, rerunfailures-10.2, shard-0.1.2, xdist-2.5.0, xdoctest-1.0.2
2022-12-01T10:52:09.1278155Z collecting ... collected 3 items
2022-12-01T10:52:09.1278614Z Running 3 items in this shard: test/distributed/pipeline/sync/skip/test_api.py::test_namespace_difference, test/distributed/pipeline/sync/skip/test_api.py::test_namespace_copy, test/distributed/pipeline/sync/skip/test_api.py::test_skippable_repr
2022-12-01T10:52:09.1278996Z 
2022-12-01T10:52:09.1279160Z distributed/pipeline/sync/skip/test_api.py::test_namespace_difference PASSED [ 33%]
2022-12-01T10:52:09.1279633Z distributed/pipeline/sync/skip/test_api.py::test_namespace_copy PASSED   [ 66%]
2022-12-01T10:52:09.1280237Z distributed/pipeline/sync/skip/test_api.py::test_skippable_repr PASSED   [100%]
2022-12-01T10:52:09.1280507Z 
2022-12-01T10:52:09.1280620Z ============================== 3 passed in 0.07s ===============================
2022-12-01T10:52:09.1280762Z 
2022-12-01T10:52:09.1281010Z ##[endgroup]
2022-12-01T10:52:09.1281509Z FINISHED PRINTING LOG FILE of distributed/pipeline/sync/skip/test_api (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-skip-test_api_n_smlu88)
2022-12-01T10:52:09.1281785Z 
2022-12-01T10:52:09.1282033Z Running distributed/pipeline/sync/skip/test_inspect_skip_layout ... [2022-12-01 10:52:09.127512]
2022-12-01T10:52:09.1282537Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/skip/test_inspect_skip_layout.py', '-v'] ... [2022-12-01 10:52:09.127843]
2022-12-01T10:52:10.5053507Z 
2022-12-01T10:52:10.5054138Z Expand the folded group to see the log file of distributed/pipeline/sync/skip/test_inspect_skip_layout
2022-12-01T10:52:10.5055584Z ##[group]PRINTING LOG FILE of distributed/pipeline/sync/skip/test_inspect_skip_layout (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-skip-test_inspect_skip_layout_dgucamt1)
2022-12-01T10:52:10.5056389Z ============================= test session starts ==============================
2022-12-01T10:52:10.5057108Z platform linux -- Python 3.7.13, pytest-7.1.3, pluggy-1.0.0 -- /opt/conda/bin/python
2022-12-01T10:52:10.5057468Z cachedir: .pytest_cache
2022-12-01T10:52:10.5058172Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples')
2022-12-01T10:52:10.5058706Z torch: 1.13.0a0+gitc13d400
2022-12-01T10:52:10.5059074Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini
2022-12-01T10:52:10.5059751Z plugins: hypothesis-5.35.1, forked-1.4.0, rerunfailures-10.2, shard-0.1.2, xdist-2.5.0, xdoctest-1.0.2
2022-12-01T10:52:10.5060206Z collecting ... collected 6 items
2022-12-01T10:52:10.5061624Z Running 6 items in this shard: test/distributed/pipeline/sync/skip/test_inspect_skip_layout.py::test_no_skippables, test/distributed/pipeline/sync/skip/test_inspect_skip_layout.py::test_inner_partition, test/distributed/pipeline/sync/skip/test_inspect_skip_layout.py::test_adjoining_partitions, test/distributed/pipeline/sync/skip/test_inspect_skip_layout.py::test_far_partitions, test/distributed/pipeline/sync/skip/test_inspect_skip_layout.py::test_pop_2_from_different_partitions, test/distributed/pipeline/sync/skip/test_inspect_skip_layout.py::test_namespace
2022-12-01T10:52:10.5062643Z 
2022-12-01T10:52:10.5062941Z distributed/pipeline/sync/skip/test_inspect_skip_layout.py::test_no_skippables PASSED [ 16%]
2022-12-01T10:52:10.5063477Z distributed/pipeline/sync/skip/test_inspect_skip_layout.py::test_inner_partition PASSED [ 33%]
2022-12-01T10:52:10.5063987Z distributed/pipeline/sync/skip/test_inspect_skip_layout.py::test_adjoining_partitions PASSED [ 50%]
2022-12-01T10:52:10.5064469Z distributed/pipeline/sync/skip/test_inspect_skip_layout.py::test_far_partitions PASSED [ 66%]
2022-12-01T10:52:10.5065210Z distributed/pipeline/sync/skip/test_inspect_skip_layout.py::test_pop_2_from_different_partitions PASSED [ 83%]
2022-12-01T10:52:10.5065838Z distributed/pipeline/sync/skip/test_inspect_skip_layout.py::test_namespace PASSED [100%]
2022-12-01T10:52:10.5066102Z 
2022-12-01T10:52:10.5066257Z ============================== 6 passed in 0.03s ===============================
2022-12-01T10:52:10.5066447Z 
2022-12-01T10:52:10.5066788Z ##[endgroup]
2022-12-01T10:52:10.5067612Z FINISHED PRINTING LOG FILE of distributed/pipeline/sync/skip/test_inspect_skip_layout (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-skip-test_inspect_skip_layout_dgucamt1)
2022-12-01T10:52:10.5068062Z 
2022-12-01T10:52:10.5068391Z Running distributed/pipeline/sync/skip/test_portal ... [2022-12-01 10:52:10.505710]
2022-12-01T10:52:10.5069117Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/skip/test_portal.py', '-v'] ... [2022-12-01 10:52:10.506134]
2022-12-01T10:52:11.9097965Z 
2022-12-01T10:52:11.9098579Z Expand the folded group to see the log file of distributed/pipeline/sync/skip/test_portal
2022-12-01T10:52:11.9099851Z ##[group]PRINTING LOG FILE of distributed/pipeline/sync/skip/test_portal (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-skip-test_portal_hgzogz7y)
2022-12-01T10:52:11.9100460Z ============================= test session starts ==============================
2022-12-01T10:52:11.9100946Z platform linux -- Python 3.7.13, pytest-7.1.3, pluggy-1.0.0 -- /opt/conda/bin/python
2022-12-01T10:52:11.9101197Z cachedir: .pytest_cache
2022-12-01T10:52:11.9101635Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples')
2022-12-01T10:52:11.9102009Z torch: 1.13.0a0+gitc13d400
2022-12-01T10:52:11.9102239Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini
2022-12-01T10:52:11.9102663Z plugins: hypothesis-5.35.1, forked-1.4.0, rerunfailures-10.2, shard-0.1.2, xdist-2.5.0, xdoctest-1.0.2
2022-12-01T10:52:11.9102956Z collecting ... collected 10 items
2022-12-01T10:52:11.9104014Z Running 10 items in this shard: test/distributed/pipeline/sync/skip/test_portal.py::test_copy_returns_on_next_device, test/distributed/pipeline/sync/skip/test_portal.py::test_blue_orange, test/distributed/pipeline/sync/skip/test_portal.py::test_blue_orange_not_requires_grad, test/distributed/pipeline/sync/skip/test_portal.py::test_use_grad, test/distributed/pipeline/sync/skip/test_portal.py::TestTensorLife::test_tensor_life_0, test/distributed/pipeline/sync/skip/test_portal.py::TestTensorLife::test_tensor_life_1, test/distributed/pipeline/sync/skip/test_portal.py::TestTensorLife::test_tensor_life_2, test/distributed/pipeline/sync/skip/test_portal.py::TestTensorLife::test_tensor_life_3, test/distributed/pipeline/sync/skip/test_portal.py::TestTensorLife::test_tensor_life_4, test/distributed/pipeline/sync/skip/test_portal.py::TestTensorLife::test_tensor_life_3_plus_1
2022-12-01T10:52:11.9104987Z 
2022-12-01T10:52:11.9105221Z distributed/pipeline/sync/skip/test_portal.py::test_copy_returns_on_next_device SKIPPED [ 10%]
2022-12-01T10:52:11.9105574Z distributed/pipeline/sync/skip/test_portal.py::test_blue_orange PASSED   [ 20%]
2022-12-01T10:52:11.9105926Z distributed/pipeline/sync/skip/test_portal.py::test_blue_orange_not_requires_grad PASSED [ 30%]
2022-12-01T10:52:11.9106255Z distributed/pipeline/sync/skip/test_portal.py::test_use_grad PASSED      [ 40%]
2022-12-01T10:52:11.9106596Z distributed/pipeline/sync/skip/test_portal.py::TestTensorLife::test_tensor_life_0 PASSED [ 50%]
2022-12-01T10:52:11.9106955Z distributed/pipeline/sync/skip/test_portal.py::TestTensorLife::test_tensor_life_1 PASSED [ 60%]
2022-12-01T10:52:11.9107295Z distributed/pipeline/sync/skip/test_portal.py::TestTensorLife::test_tensor_life_2 PASSED [ 70%]
2022-12-01T10:52:11.9107646Z distributed/pipeline/sync/skip/test_portal.py::TestTensorLife::test_tensor_life_3 PASSED [ 80%]
2022-12-01T10:52:11.9108189Z distributed/pipeline/sync/skip/test_portal.py::TestTensorLife::test_tensor_life_4 PASSED [ 90%]
2022-12-01T10:52:11.9108558Z distributed/pipeline/sync/skip/test_portal.py::TestTensorLife::test_tensor_life_3_plus_1 PASSED [100%]
2022-12-01T10:52:11.9108804Z 
2022-12-01T10:52:11.9108926Z ========================= 9 passed, 1 skipped in 0.05s =========================
2022-12-01T10:52:11.9109073Z 
2022-12-01T10:52:11.9109323Z ##[endgroup]
2022-12-01T10:52:11.9109840Z FINISHED PRINTING LOG FILE of distributed/pipeline/sync/skip/test_portal (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-skip-test_portal_hgzogz7y)
2022-12-01T10:52:11.9110140Z 
2022-12-01T10:52:11.9110355Z Running distributed/pipeline/sync/skip/test_tracker ... [2022-12-01 10:52:11.910109]
2022-12-01T10:52:11.9110832Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/skip/test_tracker.py', '-v'] ... [2022-12-01 10:52:11.910443]
2022-12-01T10:52:13.2787568Z 
2022-12-01T10:52:13.2788187Z Expand the folded group to see the log file of distributed/pipeline/sync/skip/test_tracker
2022-12-01T10:52:13.2789399Z ##[group]PRINTING LOG FILE of distributed/pipeline/sync/skip/test_tracker (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-skip-test_tracker_d5y9zalr)
2022-12-01T10:52:13.2789848Z ============================= test session starts ==============================
2022-12-01T10:52:13.2790322Z platform linux -- Python 3.7.13, pytest-7.1.3, pluggy-1.0.0 -- /opt/conda/bin/python
2022-12-01T10:52:13.2790568Z cachedir: .pytest_cache
2022-12-01T10:52:13.2791005Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples')
2022-12-01T10:52:13.2791348Z torch: 1.13.0a0+gitc13d400
2022-12-01T10:52:13.2791703Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini
2022-12-01T10:52:13.2792111Z plugins: hypothesis-5.35.1, forked-1.4.0, rerunfailures-10.2, shard-0.1.2, xdist-2.5.0, xdoctest-1.0.2
2022-12-01T10:52:13.2792401Z collecting ... collected 6 items
2022-12-01T10:52:13.2793143Z Running 6 items in this shard: test/distributed/pipeline/sync/skip/test_tracker.py::test_default_skip_tracker, test/distributed/pipeline/sync/skip/test_tracker.py::test_default_skip_tracker_by_data_parallel, test/distributed/pipeline/sync/skip/test_tracker.py::test_reuse_portal, test/distributed/pipeline/sync/skip/test_tracker.py::test_no_copy_no_portal, test/distributed/pipeline/sync/skip/test_tracker.py::test_tensor_life_without_checkpointing, test/distributed/pipeline/sync/skip/test_tracker.py::test_tensor_life_with_checkpointing
2022-12-01T10:52:13.2793754Z 
2022-12-01T10:52:13.2793920Z distributed/pipeline/sync/skip/test_tracker.py::test_default_skip_tracker PASSED [ 16%]
2022-12-01T10:52:13.2794278Z distributed/pipeline/sync/skip/test_tracker.py::test_default_skip_tracker_by_data_parallel SKIPPED [ 33%]
2022-12-01T10:52:13.2794680Z distributed/pipeline/sync/skip/test_tracker.py::test_reuse_portal PASSED [ 50%]
2022-12-01T10:52:13.2795144Z distributed/pipeline/sync/skip/test_tracker.py::test_no_copy_no_portal PASSED [ 66%]
2022-12-01T10:52:13.2795507Z distributed/pipeline/sync/skip/test_tracker.py::test_tensor_life_without_checkpointing PASSED [ 83%]
2022-12-01T10:52:13.2795873Z distributed/pipeline/sync/skip/test_tracker.py::test_tensor_life_with_checkpointing PASSED [100%]
2022-12-01T10:52:13.2796123Z 
2022-12-01T10:52:13.2796241Z ========================= 5 passed, 1 skipped in 0.03s =========================
2022-12-01T10:52:13.2796386Z 
2022-12-01T10:52:13.2796612Z ##[endgroup]
2022-12-01T10:52:13.2797165Z FINISHED PRINTING LOG FILE of distributed/pipeline/sync/skip/test_tracker (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-skip-test_tracker_d5y9zalr)
2022-12-01T10:52:13.2797476Z 
2022-12-01T10:52:13.2797727Z Running distributed/pipeline/sync/test_balance ... [2022-12-01 10:52:13.279085]
2022-12-01T10:52:13.2798187Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/test_balance.py', '-v'] ... [2022-12-01 10:52:13.279414]
2022-12-01T10:52:18.6945733Z 
2022-12-01T10:52:18.6946267Z Expand the folded group to see the log file of distributed/pipeline/sync/test_balance
2022-12-01T10:52:18.6947657Z ##[group]PRINTING LOG FILE of distributed/pipeline/sync/test_balance (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-test_balance_srqs3oe9)
2022-12-01T10:52:18.6948415Z ============================= test session starts ==============================
2022-12-01T10:52:18.6949057Z platform linux -- Python 3.7.13, pytest-7.1.3, pluggy-1.0.0 -- /opt/conda/bin/python
2022-12-01T10:52:18.6949345Z cachedir: .pytest_cache
2022-12-01T10:52:18.6949967Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples')
2022-12-01T10:52:18.6950473Z torch: 1.13.0a0+gitc13d400
2022-12-01T10:52:18.6950935Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini
2022-12-01T10:52:18.6951441Z plugins: hypothesis-5.35.1, forked-1.4.0, rerunfailures-10.2, shard-0.1.2, xdist-2.5.0, xdoctest-1.0.2
2022-12-01T10:52:18.6951730Z collecting ... collected 15 items
2022-12-01T10:52:18.6953362Z Running 15 items in this shard: test/distributed/pipeline/sync/test_balance.py::test_blockpartition, test/distributed/pipeline/sync/test_balance.py::test_blockpartition_zeros, test/distributed/pipeline/sync/test_balance.py::test_blockpartition_non_positive_partitions, test/distributed/pipeline/sync/test_balance.py::test_blockpartition_short_sequence, test/distributed/pipeline/sync/test_balance.py::test_balance_by_time[cpu], test/distributed/pipeline/sync/test_balance.py::test_balance_by_time_loop_resets_input, test/distributed/pipeline/sync/test_balance.py::test_balance_by_size_latent, test/distributed/pipeline/sync/test_balance.py::test_balance_by_size_param, test/distributed/pipeline/sync/test_balance.py::test_balance_by_size_param_scale, test/distributed/pipeline/sync/test_balance.py::test_layerwise_sandbox[cpu], test/distributed/pipeline/sync/test_balance.py::test_sandbox_during_profiling[cpu], test/distributed/pipeline/sync/test_balance.py::test_not_training, test/distributed/pipeline/sync/test_balance.py::test_balance_by_time_tuple, test/distributed/pipeline/sync/test_balance.py::test_balance_by_size_tuple, test/distributed/pipeline/sync/test_balance.py::test_already_has_grad
2022-12-01T10:52:18.6954744Z 
2022-12-01T10:52:18.6954907Z distributed/pipeline/sync/test_balance.py::test_blockpartition PASSED    [  6%]
2022-12-01T10:52:18.6955234Z distributed/pipeline/sync/test_balance.py::test_blockpartition_zeros PASSED [ 13%]
2022-12-01T10:52:18.6955601Z distributed/pipeline/sync/test_balance.py::test_blockpartition_non_positive_partitions PASSED [ 20%]
2022-12-01T10:52:18.6955972Z distributed/pipeline/sync/test_balance.py::test_blockpartition_short_sequence PASSED [ 26%]
2022-12-01T10:52:18.6956307Z distributed/pipeline/sync/test_balance.py::test_balance_by_time[cpu] SKIPPED [ 33%]
2022-12-01T10:52:18.6956652Z distributed/pipeline/sync/test_balance.py::test_balance_by_time_loop_resets_input PASSED [ 40%]
2022-12-01T10:52:18.6957003Z distributed/pipeline/sync/test_balance.py::test_balance_by_size_latent SKIPPED [ 46%]
2022-12-01T10:52:18.6957340Z distributed/pipeline/sync/test_balance.py::test_balance_by_size_param SKIPPED [ 53%]
2022-12-01T10:52:18.6957670Z distributed/pipeline/sync/test_balance.py::test_balance_by_size_param_scale SKIPPED [ 60%]
2022-12-01T10:52:18.6958014Z distributed/pipeline/sync/test_balance.py::test_layerwise_sandbox[cpu] PASSED [ 66%]
2022-12-01T10:52:18.6958358Z distributed/pipeline/sync/test_balance.py::test_sandbox_during_profiling[cpu] PASSED [ 73%]
2022-12-01T10:52:18.6958686Z distributed/pipeline/sync/test_balance.py::test_not_training PASSED      [ 80%]
2022-12-01T10:52:18.6959010Z distributed/pipeline/sync/test_balance.py::test_balance_by_time_tuple PASSED [ 86%]
2022-12-01T10:52:18.6959345Z distributed/pipeline/sync/test_balance.py::test_balance_by_size_tuple SKIPPED [ 93%]
2022-12-01T10:52:18.6959676Z distributed/pipeline/sync/test_balance.py::test_already_has_grad PASSED  [100%]
2022-12-01T10:52:18.6959928Z 
2022-12-01T10:52:18.6960049Z ======================== 10 passed, 5 skipped in 4.06s =========================
2022-12-01T10:52:18.6960240Z 
2022-12-01T10:52:18.6960511Z ##[endgroup]
2022-12-01T10:52:18.6961025Z FINISHED PRINTING LOG FILE of distributed/pipeline/sync/test_balance (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-test_balance_srqs3oe9)
2022-12-01T10:52:18.6961307Z 
2022-12-01T10:52:18.6961510Z Running distributed/pipeline/sync/test_checkpoint ... [2022-12-01 10:52:18.694779]
2022-12-01T10:52:18.6961982Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/test_checkpoint.py', '-v'] ... [2022-12-01 10:52:18.695161]
2022-12-01T10:52:20.0960931Z 
2022-12-01T10:52:20.0961544Z Expand the folded group to see the log file of distributed/pipeline/sync/test_checkpoint
2022-12-01T10:52:20.0965601Z ##[group]PRINTING LOG FILE of distributed/pipeline/sync/test_checkpoint (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-test_checkpoint_ukup1ixb)
2022-12-01T10:52:20.0968825Z ============================= test session starts ==============================
2022-12-01T10:52:20.0969535Z platform linux -- Python 3.7.13, pytest-7.1.3, pluggy-1.0.0 -- /opt/conda/bin/python
2022-12-01T10:52:20.0969925Z cachedir: .pytest_cache
2022-12-01T10:52:20.0970589Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples')
2022-12-01T10:52:20.0971067Z torch: 1.13.0a0+gitc13d400
2022-12-01T10:52:20.0971416Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini
2022-12-01T10:52:20.0972146Z plugins: hypothesis-5.35.1, forked-1.4.0, rerunfailures-10.2, shard-0.1.2, xdist-2.5.0, xdoctest-1.0.2
2022-12-01T10:52:20.0972586Z collecting ... collected 7 items
2022-12-01T10:52:20.0973858Z Running 7 items in this shard: test/distributed/pipeline/sync/test_checkpoint.py::test_serial_checkpoints[cpu], test/distributed/pipeline/sync/test_checkpoint.py::test_not_requires_grad, test/distributed/pipeline/sync/test_checkpoint.py::test_not_requires_grad_with_parameter, test/distributed/pipeline/sync/test_checkpoint.py::test_random_in_checkpoint[cpu], test/distributed/pipeline/sync/test_checkpoint.py::test_detect_checkpointing_recomputing, test/distributed/pipeline/sync/test_checkpoint.py::test_detect_checkpointing_recomputing_without_checkpoint, test/distributed/pipeline/sync/test_checkpoint.py::test_non_grad_output
2022-12-01T10:52:20.0974975Z 
2022-12-01T10:52:20.0975243Z distributed/pipeline/sync/test_checkpoint.py::test_serial_checkpoints[cpu] PASSED [ 14%]
2022-12-01T10:52:20.0975802Z distributed/pipeline/sync/test_checkpoint.py::test_not_requires_grad PASSED [ 28%]
2022-12-01T10:52:20.0976341Z distributed/pipeline/sync/test_checkpoint.py::test_not_requires_grad_with_parameter PASSED [ 42%]
2022-12-01T10:52:20.0976875Z distributed/pipeline/sync/test_checkpoint.py::test_random_in_checkpoint[cpu] PASSED [ 57%]
2022-12-01T10:52:20.0977505Z distributed/pipeline/sync/test_checkpoint.py::test_detect_checkpointing_recomputing PASSED [ 71%]
2022-12-01T10:52:20.0978230Z distributed/pipeline/sync/test_checkpoint.py::test_detect_checkpointing_recomputing_without_checkpoint PASSED [ 85%]
2022-12-01T10:52:20.0978891Z distributed/pipeline/sync/test_checkpoint.py::test_non_grad_output PASSED [100%]
2022-12-01T10:52:20.0979192Z 
2022-12-01T10:52:20.0979379Z ============================== 7 passed in 0.04s ===============================
2022-12-01T10:52:20.0979606Z 
2022-12-01T10:52:20.0980028Z ##[endgroup]
2022-12-01T10:52:20.0993446Z FINISHED PRINTING LOG FILE of distributed/pipeline/sync/test_checkpoint (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-test_checkpoint_ukup1ixb)
2022-12-01T10:52:20.0993863Z 
2022-12-01T10:52:20.0994113Z Running distributed/pipeline/sync/test_deferred_batch_norm ... [2022-12-01 10:52:20.096408]
2022-12-01T10:52:20.0994858Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/test_deferred_batch_norm.py', '-v'] ... [2022-12-01 10:52:20.096849]
2022-12-01T10:52:22.0957959Z 
2022-12-01T10:52:22.0958589Z Expand the folded group to see the log file of distributed/pipeline/sync/test_deferred_batch_norm
2022-12-01T10:52:22.0959852Z ##[group]PRINTING LOG FILE of distributed/pipeline/sync/test_deferred_batch_norm (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-test_deferred_batch_norm_qlt9qt6f)
2022-12-01T10:52:22.0960647Z ============================= test session starts ==============================
2022-12-01T10:52:22.0961426Z platform linux -- Python 3.7.13, pytest-7.1.3, pluggy-1.0.0 -- /opt/conda/bin/python
2022-12-01T10:52:22.0961883Z cachedir: .pytest_cache
2022-12-01T10:52:22.0962659Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples')
2022-12-01T10:52:22.0962974Z torch: 1.13.0a0+gitc13d400
2022-12-01T10:52:22.0963250Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini
2022-12-01T10:52:22.0963682Z plugins: hypothesis-5.35.1, forked-1.4.0, rerunfailures-10.2, shard-0.1.2, xdist-2.5.0, xdoctest-1.0.2
2022-12-01T10:52:22.0963966Z collecting ... collected 11 items
2022-12-01T10:52:22.0966087Z Running 11 items in this shard: test/distributed/pipeline/sync/test_deferred_batch_norm.py::test_transparency[True-1], test/distributed/pipeline/sync/test_deferred_batch_norm.py::test_transparency[True-4], test/distributed/pipeline/sync/test_deferred_batch_norm.py::test_transparency[False-1], test/distributed/pipeline/sync/test_deferred_batch_norm.py::test_transparency[False-4], test/distributed/pipeline/sync/test_deferred_batch_norm.py::test_running_stats[0.1], test/distributed/pipeline/sync/test_deferred_batch_norm.py::test_running_stats[None], test/distributed/pipeline/sync/test_deferred_batch_norm.py::test_convert_deferred_batch_norm, test/distributed/pipeline/sync/test_deferred_batch_norm.py::test_eval, test/distributed/pipeline/sync/test_deferred_batch_norm.py::test_optimize, test/distributed/pipeline/sync/test_deferred_batch_norm.py::test_conv_bn, test/distributed/pipeline/sync/test_deferred_batch_norm.py::test_input_requiring_grad
2022-12-01T10:52:22.0967092Z 
2022-12-01T10:52:22.0967345Z distributed/pipeline/sync/test_deferred_batch_norm.py::test_transparency[True-1] PASSED [  9%]
2022-12-01T10:52:22.0967764Z distributed/pipeline/sync/test_deferred_batch_norm.py::test_transparency[True-4] PASSED [ 18%]
2022-12-01T10:52:22.0968194Z distributed/pipeline/sync/test_deferred_batch_norm.py::test_transparency[False-1] PASSED [ 27%]
2022-12-01T10:52:22.0968628Z distributed/pipeline/sync/test_deferred_batch_norm.py::test_transparency[False-4] PASSED [ 36%]
2022-12-01T10:52:22.0968971Z distributed/pipeline/sync/test_deferred_batch_norm.py::test_running_stats[0.1] PASSED [ 45%]
2022-12-01T10:52:22.0969319Z distributed/pipeline/sync/test_deferred_batch_norm.py::test_running_stats[None] PASSED [ 54%]
2022-12-01T10:52:22.0969680Z distributed/pipeline/sync/test_deferred_batch_norm.py::test_convert_deferred_batch_norm PASSED [ 63%]
2022-12-01T10:52:22.0970032Z distributed/pipeline/sync/test_deferred_batch_norm.py::test_eval PASSED  [ 72%]
2022-12-01T10:52:22.0970353Z distributed/pipeline/sync/test_deferred_batch_norm.py::test_optimize PASSED [ 81%]
2022-12-01T10:52:22.0970689Z distributed/pipeline/sync/test_deferred_batch_norm.py::test_conv_bn PASSED [ 90%]
2022-12-01T10:52:22.0971030Z distributed/pipeline/sync/test_deferred_batch_norm.py::test_input_requiring_grad PASSED [100%]
2022-12-01T10:52:22.0971220Z 
2022-12-01T10:52:22.0971332Z ============================== 11 passed in 0.65s ==============================
2022-12-01T10:52:22.0971459Z 
2022-12-01T10:52:22.0971691Z ##[endgroup]
2022-12-01T10:52:22.0972285Z FINISHED PRINTING LOG FILE of distributed/pipeline/sync/test_deferred_batch_norm (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-test_deferred_batch_norm_qlt9qt6f)
2022-12-01T10:52:22.0972591Z 
2022-12-01T10:52:22.0972806Z Running distributed/pipeline/sync/test_inplace ... [2022-12-01 10:52:22.096020]
2022-12-01T10:52:22.0973465Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/test_inplace.py', '-v'] ... [2022-12-01 10:52:22.096374]
2022-12-01T10:52:23.5748027Z 
2022-12-01T10:52:23.5748660Z Expand the folded group to see the log file of distributed/pipeline/sync/test_inplace
2022-12-01T10:52:23.5749637Z ##[group]PRINTING LOG FILE of distributed/pipeline/sync/test_inplace (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-test_inplace_x1n6je71)
2022-12-01T10:52:23.5750065Z ============================= test session starts ==============================
2022-12-01T10:52:23.5750532Z platform linux -- Python 3.7.13, pytest-7.1.3, pluggy-1.0.0 -- /opt/conda/bin/python
2022-12-01T10:52:23.5750796Z cachedir: .pytest_cache
2022-12-01T10:52:23.5751233Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples')
2022-12-01T10:52:23.5751562Z torch: 1.13.0a0+gitc13d400
2022-12-01T10:52:23.5751808Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini
2022-12-01T10:52:23.5752244Z plugins: hypothesis-5.35.1, forked-1.4.0, rerunfailures-10.2, shard-0.1.2, xdist-2.5.0, xdoctest-1.0.2
2022-12-01T10:52:23.5752537Z collecting ... collected 3 items
2022-12-01T10:52:23.5753003Z Running 3 items in this shard: test/distributed/pipeline/sync/test_inplace.py::test_inplace_on_requires_grad, test/distributed/pipeline/sync/test_inplace.py::test_inplace_on_not_requires_grad, test/distributed/pipeline/sync/test_inplace.py::test_inplace_incorrect_grad
2022-12-01T10:52:23.5753381Z 
2022-12-01T10:52:23.5753532Z distributed/pipeline/sync/test_inplace.py::test_inplace_on_requires_grad PASSED [ 33%]
2022-12-01T10:52:23.5753885Z distributed/pipeline/sync/test_inplace.py::test_inplace_on_not_requires_grad XFAIL [ 66%]
2022-12-01T10:52:23.5754231Z distributed/pipeline/sync/test_inplace.py::test_inplace_incorrect_grad XFAIL [100%]
2022-12-01T10:52:23.5754417Z 
2022-12-01T10:52:23.5754539Z ========================= 1 passed, 2 xfailed in 0.11s =========================
2022-12-01T10:52:23.5754689Z 
2022-12-01T10:52:23.5754896Z ##[endgroup]
2022-12-01T10:52:23.5755469Z FINISHED PRINTING LOG FILE of distributed/pipeline/sync/test_inplace (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-test_inplace_x1n6je71)
2022-12-01T10:52:23.5755756Z 
2022-12-01T10:52:23.5755964Z Running distributed/pipeline/sync/test_phony ... [2022-12-01 10:52:23.575052]
2022-12-01T10:52:23.5756406Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/test_phony.py', '-v'] ... [2022-12-01 10:52:23.575424]
2022-12-01T10:52:24.9457418Z 
2022-12-01T10:52:24.9457976Z Expand the folded group to see the log file of distributed/pipeline/sync/test_phony
2022-12-01T10:52:24.9458874Z ##[group]PRINTING LOG FILE of distributed/pipeline/sync/test_phony (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-test_phony_f1n2_amm)
2022-12-01T10:52:24.9459364Z ============================= test session starts ==============================
2022-12-01T10:52:24.9459888Z platform linux -- Python 3.7.13, pytest-7.1.3, pluggy-1.0.0 -- /opt/conda/bin/python
2022-12-01T10:52:24.9460163Z cachedir: .pytest_cache
2022-12-01T10:52:24.9460885Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples')
2022-12-01T10:52:24.9461214Z torch: 1.13.0a0+gitc13d400
2022-12-01T10:52:24.9461440Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini
2022-12-01T10:52:24.9461859Z plugins: hypothesis-5.35.1, forked-1.4.0, rerunfailures-10.2, shard-0.1.2, xdist-2.5.0, xdoctest-1.0.2
2022-12-01T10:52:24.9462147Z collecting ... collected 4 items
2022-12-01T10:52:24.9462882Z Running 4 items in this shard: test/distributed/pipeline/sync/test_phony.py::test_phony_size, test/distributed/pipeline/sync/test_phony.py::test_phony_requires_grad, test/distributed/pipeline/sync/test_phony.py::test_cached_phony, test/distributed/pipeline/sync/test_phony.py::test_phony_in_autograd_function
2022-12-01T10:52:24.9463277Z 
2022-12-01T10:52:24.9463492Z distributed/pipeline/sync/test_phony.py::test_phony_size PASSED          [ 25%]
2022-12-01T10:52:24.9463895Z distributed/pipeline/sync/test_phony.py::test_phony_requires_grad PASSED [ 50%]
2022-12-01T10:52:24.9464239Z distributed/pipeline/sync/test_phony.py::test_cached_phony PASSED        [ 75%]
2022-12-01T10:52:24.9464575Z distributed/pipeline/sync/test_phony.py::test_phony_in_autograd_function PASSED [100%]
2022-12-01T10:52:24.9464777Z 
2022-12-01T10:52:24.9464906Z ============================== 4 passed in 0.03s ===============================
2022-12-01T10:52:24.9465031Z 
2022-12-01T10:52:24.9465284Z ##[endgroup]
2022-12-01T10:52:24.9465770Z FINISHED PRINTING LOG FILE of distributed/pipeline/sync/test_phony (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-test_phony_f1n2_amm)
2022-12-01T10:52:24.9466047Z 
2022-12-01T10:52:24.9466269Z Running distributed/pipeline/sync/test_pipeline ... [2022-12-01 10:52:24.945966]
2022-12-01T10:52:24.9466721Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/test_pipeline.py', '-v'] ... [2022-12-01 10:52:24.946330]
2022-12-01T10:52:26.3058458Z 
2022-12-01T10:52:26.3059110Z Expand the folded group to see the log file of distributed/pipeline/sync/test_pipeline
2022-12-01T10:52:26.3060022Z ##[group]PRINTING LOG FILE of distributed/pipeline/sync/test_pipeline (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-test_pipeline_hq6uz5wf)
2022-12-01T10:52:26.3060592Z ============================= test session starts ==============================
2022-12-01T10:52:26.3061077Z platform linux -- Python 3.7.13, pytest-7.1.3, pluggy-1.0.0 -- /opt/conda/bin/python
2022-12-01T10:52:26.3061329Z cachedir: .pytest_cache
2022-12-01T10:52:26.3061793Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples')
2022-12-01T10:52:26.3062245Z torch: 1.13.0a0+gitc13d400
2022-12-01T10:52:26.3062504Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini
2022-12-01T10:52:26.3062917Z plugins: hypothesis-5.35.1, forked-1.4.0, rerunfailures-10.2, shard-0.1.2, xdist-2.5.0, xdoctest-1.0.2
2022-12-01T10:52:26.3063202Z collecting ... collected 1 item
2022-12-01T10:52:26.3063497Z Running 1 items in this shard: test/distributed/pipeline/sync/test_pipeline.py::test_clock_cycles
2022-12-01T10:52:26.3063692Z 
2022-12-01T10:52:26.3063835Z distributed/pipeline/sync/test_pipeline.py::test_clock_cycles PASSED     [100%]
2022-12-01T10:52:26.3064017Z 
2022-12-01T10:52:26.3064125Z ============================== 1 passed in 0.02s ===============================
2022-12-01T10:52:26.3064264Z 
2022-12-01T10:52:26.3064497Z ##[endgroup]
2022-12-01T10:52:26.3064971Z FINISHED PRINTING LOG FILE of distributed/pipeline/sync/test_pipeline (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-test_pipeline_hq6uz5wf)
2022-12-01T10:52:26.3065304Z 
2022-12-01T10:52:26.3065569Z Running distributed/pipeline/sync/test_transparency ... [2022-12-01 10:52:26.306111]
2022-12-01T10:52:26.3066056Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/test_transparency.py', '-v'] ... [2022-12-01 10:52:26.306433]
2022-12-01T10:52:27.7339535Z 
2022-12-01T10:52:27.7340103Z Expand the folded group to see the log file of distributed/pipeline/sync/test_transparency
2022-12-01T10:52:27.7341291Z ##[group]PRINTING LOG FILE of distributed/pipeline/sync/test_transparency (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-test_transparency_ggrf9zpj)
2022-12-01T10:52:27.7341733Z ============================= test session starts ==============================
2022-12-01T10:52:27.7342211Z platform linux -- Python 3.7.13, pytest-7.1.3, pluggy-1.0.0 -- /opt/conda/bin/python
2022-12-01T10:52:27.7342475Z cachedir: .pytest_cache
2022-12-01T10:52:27.7342895Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples')
2022-12-01T10:52:27.7343430Z torch: 1.13.0a0+gitc13d400
2022-12-01T10:52:27.7343799Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini
2022-12-01T10:52:27.7344257Z plugins: hypothesis-5.35.1, forked-1.4.0, rerunfailures-10.2, shard-0.1.2, xdist-2.5.0, xdoctest-1.0.2
2022-12-01T10:52:27.7344532Z collecting ... collected 1 item
2022-12-01T10:52:27.7344840Z Running 1 items in this shard: test/distributed/pipeline/sync/test_transparency.py::test_simple_linears
2022-12-01T10:52:27.7345042Z 
2022-12-01T10:52:27.7345209Z distributed/pipeline/sync/test_transparency.py::test_simple_linears PASSED [100%]
2022-12-01T10:52:27.7345399Z 
2022-12-01T10:52:27.7345496Z ============================== 1 passed in 0.07s ===============================
2022-12-01T10:52:27.7345639Z 
2022-12-01T10:52:27.7345875Z ##[endgroup]
2022-12-01T10:52:27.7346380Z FINISHED PRINTING LOG FILE of distributed/pipeline/sync/test_transparency (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-test_transparency_ggrf9zpj)
2022-12-01T10:52:27.7346818Z 
2022-12-01T10:52:27.7347039Z Running distributed/test_nccl ... [2022-12-01 10:52:27.734254]
2022-12-01T10:52:27.7347528Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_nccl.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-12-01 10:52:27.734584]
2022-12-01T10:52:29.1967977Z 
2022-12-01T10:52:29.1968524Z Expand the folded group to see the log file of distributed/test_nccl
2022-12-01T10:52:29.1969631Z ##[group]PRINTING LOG FILE of distributed/test_nccl (/var/lib/jenkins/workspace/test/test-reports/distributed-test_nccl_aw4ctejt)
2022-12-01T10:52:29.1970300Z CUDA not available, skipping tests
2022-12-01T10:52:29.1970551Z 
2022-12-01T10:52:29.1970662Z Running tests...
2022-12-01T10:52:29.1971068Z ----------------------------------------------------------------------
2022-12-01T10:52:29.1971241Z 
2022-12-01T10:52:29.1971441Z ----------------------------------------------------------------------
2022-12-01T10:52:29.1971683Z Ran 0 tests in 0.000s
2022-12-01T10:52:29.1971872Z 
2022-12-01T10:52:29.1971936Z OK
2022-12-01T10:52:29.1972034Z 
2022-12-01T10:52:29.1972119Z Generating XML reports...
2022-12-01T10:52:29.1972469Z Test results will be stored in test-reports/python-unittest/distributed.test_nccl
2022-12-01T10:52:29.1972660Z 
2022-12-01T10:52:29.1972875Z ##[endgroup]
2022-12-01T10:52:29.1973292Z FINISHED PRINTING LOG FILE of distributed/test_nccl (/var/lib/jenkins/workspace/test/test-reports/distributed-test_nccl_aw4ctejt)
2022-12-01T10:52:29.1973522Z 
2022-12-01T10:52:29.4854057Z 
2022-12-01T10:52:29.4854397Z real	47m38.497s
2022-12-01T10:52:29.4854642Z user	86m37.327s
2022-12-01T10:52:29.4854875Z sys	19m23.295s
2022-12-01T10:52:29.4855065Z + assert_git_not_dirty
2022-12-01T10:52:29.4855456Z + [[ linux-focal-py3.7-gcc7 != *rocm* ]]
2022-12-01T10:52:29.4855849Z + [[ linux-focal-py3.7-gcc7 != *xla* ]]
2022-12-01T10:52:29.4856850Z ++ git status --porcelain
2022-12-01T10:52:30.5984089Z + git_status=
2022-12-01T10:52:30.5984596Z + [[ -n '' ]]
2022-12-01T10:52:30.5984899Z + [[ linux-focal-py3.7-gcc7 == *cuda* ]]
2022-12-01T10:52:30.5985117Z + [[ 2 == 1 ]]
2022-12-01T10:52:30.6040211Z Prepare all required actions
2022-12-01T10:52:30.6040534Z Getting action download info
2022-12-01T10:52:30.7845029Z ##[group]Run ./.github/actions/get-workflow-job-id
2022-12-01T10:52:30.7845254Z with:
2022-12-01T10:52:30.7845580Z   github-token: ***
2022-12-01T10:52:30.7845750Z env:
2022-12-01T10:52:30.7845910Z   GIT_DEFAULT_BRANCH: master
2022-12-01T10:52:30.7846096Z ##[endgroup]
2022-12-01T10:52:30.7873016Z ##[group]Run nick-fields/retry@7d4a37704547a311dbb66ebdf5b23ec19374a767
2022-12-01T10:52:30.7873255Z with:
2022-12-01T10:52:30.7873406Z   shell: bash
2022-12-01T10:52:30.7873588Z   timeout_minutes: 10
2022-12-01T10:52:30.7873774Z   max_attempts: 5
2022-12-01T10:52:30.7873945Z   retry_wait_seconds: 30
2022-12-01T10:52:30.7874332Z   command: set -eux
python3 -m pip install requests==2.26.0
GHA_WORKFLOW_JOB_ID=$(python3 .github/scripts/get_workflow_job_id.py "${GITHUB_RUN_ID}" "${RUNNER_NAME}")
echo "::set-output name=job-id::${GHA_WORKFLOW_JOB_ID}"

2022-12-01T10:52:30.7874844Z   polling_interval_seconds: 1
2022-12-01T10:52:30.7875049Z   warning_on_retry: true
2022-12-01T10:52:30.7875330Z   continue_on_error: false
2022-12-01T10:52:30.7875513Z env:
2022-12-01T10:52:30.7875673Z   GIT_DEFAULT_BRANCH: master
2022-12-01T10:52:30.7875998Z   GITHUB_TOKEN: ***
2022-12-01T10:52:30.7876179Z ##[endgroup]
2022-12-01T10:52:31.1066564Z 
2022-12-01T10:52:31.1073729Z ##[warning]The `set-output` command is deprecated and will be disabled soon. Please upgrade to using Environment Files. For more information see: https://github.blog/changelog/2022-10-11-github-actions-deprecating-save-state-and-set-output-commands/
2022-12-01T10:52:31.1112634Z + python3 -m pip install requests==2.26.0
2022-12-01T10:52:31.3228099Z Defaulting to user installation because normal site-packages is not writeable
2022-12-01T10:52:31.3412072Z Requirement already satisfied: requests==2.26.0 in /home/ec2-user/.local/lib/python3.7/site-packages (2.26.0)
2022-12-01T10:52:31.3542451Z Requirement already satisfied: certifi>=2017.4.17 in /home/ec2-user/.local/lib/python3.7/site-packages (from requests==2.26.0) (2022.9.24)
2022-12-01T10:52:31.3551167Z Requirement already satisfied: urllib3<1.27,>=1.21.1 in /home/ec2-user/.local/lib/python3.7/site-packages (from requests==2.26.0) (1.26.13)
2022-12-01T10:52:31.3705484Z Requirement already satisfied: charset-normalizer~=2.0.0; python_version >= "3" in /home/ec2-user/.local/lib/python3.7/site-packages (from requests==2.26.0) (2.0.12)
2022-12-01T10:52:31.3724822Z Requirement already satisfied: idna<4,>=2.5; python_version >= "3" in /home/ec2-user/.local/lib/python3.7/site-packages (from requests==2.26.0) (3.4)
2022-12-01T10:52:31.5637979Z ++ python3 .github/scripts/get_workflow_job_id.py 3591403534 i-0ede55ee7772c82ca
2022-12-01T10:52:34.1025512Z + GHA_WORKFLOW_JOB_ID=9818326018
2022-12-01T10:52:34.1026344Z + echo '::set-output name=job-id::9818326018'
2022-12-01T10:52:34.1034311Z ##[warning]The `set-output` command is deprecated and will be disabled soon. Please upgrade to using Environment Files. For more information see: https://github.blog/changelog/2022-10-11-github-actions-deprecating-save-state-and-set-output-commands/
2022-12-01T10:52:34.1129532Z Command completed after 1 attempt(s).
2022-12-01T10:52:34.1129729Z 
2022-12-01T10:52:34.1130967Z ##[warning]The `set-output` command is deprecated and will be disabled soon. Please upgrade to using Environment Files. For more information see: https://github.blog/changelog/2022-10-11-github-actions-deprecating-save-state-and-set-output-commands/
2022-12-01T10:52:34.1234883Z ##[group]Run kill "$MONITOR_SCRIPT_PID"
2022-12-01T10:52:34.1235128Z kill "$MONITOR_SCRIPT_PID"
2022-12-01T10:52:34.1246642Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-12-01T10:52:34.1246853Z env:
2022-12-01T10:52:34.1247036Z   GIT_DEFAULT_BRANCH: master
2022-12-01T10:52:34.1247243Z   MONITOR_SCRIPT_PID: 13641
2022-12-01T10:52:34.1247423Z ##[endgroup]
2022-12-01T10:52:34.1321822Z Prepare all required actions
2022-12-01T10:52:34.1322106Z Getting action download info
2022-12-01T10:52:34.3100628Z Download action repository 'actions/upload-artifact@v2' (SHA:82c141cc518b40d92cc801eee768e7aafc9c2fa2)
2022-12-01T10:52:34.4603519Z ##[group]Run ./.github/actions/upload-test-artifacts
2022-12-01T10:52:34.4603740Z with:
2022-12-01T10:52:34.4603978Z   file-suffix: test-distributed-2-2-linux.2xlarge_9818326018
2022-12-01T10:52:34.4604210Z env:
2022-12-01T10:52:34.4604374Z   GIT_DEFAULT_BRANCH: master
2022-12-01T10:52:34.4604564Z ##[endgroup]
2022-12-01T10:52:34.4628947Z ##[group]Run # Remove any previous test jsons if they exist
2022-12-01T10:52:34.4629231Z # Remove any previous test jsons if they exist
2022-12-01T10:52:34.4629462Z rm -f test-jsons-*.zip
2022-12-01T10:52:34.4629707Z zip -r "test-jsons-${FILE_SUFFIX}.zip" test -i '*.json'
2022-12-01T10:52:34.4640973Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-12-01T10:52:34.4641234Z env:
2022-12-01T10:52:34.4641413Z   GIT_DEFAULT_BRANCH: master
2022-12-01T10:52:34.4641761Z   FILE_SUFFIX: test-distributed-2-2-linux.2xlarge_9818326018
2022-12-01T10:52:34.4641998Z ##[endgroup]
2022-12-01T10:52:34.4744194Z   adding: test/allowlist_for_publicAPI.json (deflated 80%)
2022-12-01T10:52:34.4770742Z   adding: test/benchmark_utils/callgrind_artifacts.json (deflated 92%)
2022-12-01T10:52:34.4776632Z   adding: test/profiler/profiler_utils_mock_events.json (deflated 87%)
2022-12-01T10:52:34.4777927Z   adding: test/.pytorch-slow-tests.json (deflated 76%)
2022-12-01T10:52:34.4786437Z   adding: test/.pytorch-disabled-tests.json (deflated 86%)
2022-12-01T10:52:34.4806449Z ##[group]Run # Remove any previous test reports if they exist
2022-12-01T10:52:34.4806746Z # Remove any previous test reports if they exist
2022-12-01T10:52:34.4806984Z rm -f test-reports-*.zip
2022-12-01T10:52:34.4807233Z zip -r "test-reports-${FILE_SUFFIX}.zip" test -i '*.xml'
2022-12-01T10:52:34.4818208Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-12-01T10:52:34.4818434Z env:
2022-12-01T10:52:34.4818610Z   GIT_DEFAULT_BRANCH: master
2022-12-01T10:52:34.4818852Z   FILE_SUFFIX: test-distributed-2-2-linux.2xlarge_9818326018
2022-12-01T10:52:34.4819088Z ##[endgroup]
2022-12-01T10:52:34.4954482Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20221201100456.xml (deflated 39%)
2022-12-01T10:52:34.4955488Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20221201100459.xml (deflated 40%)
2022-12-01T10:52:34.4956438Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20221201100503.xml (deflated 39%)
2022-12-01T10:52:34.4957401Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20221201100506.xml (deflated 40%)
2022-12-01T10:52:34.4958355Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20221201100510.xml (deflated 40%)
2022-12-01T10:52:34.4959299Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20221201100513.xml (deflated 40%)
2022-12-01T10:52:34.4960231Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20221201100517.xml (deflated 40%)
2022-12-01T10:52:34.4961180Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20221201100520.xml (deflated 40%)
2022-12-01T10:52:34.4962115Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20221201100524.xml (deflated 40%)
2022-12-01T10:52:34.4963047Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20221201100527.xml (deflated 40%)
2022-12-01T10:52:34.4963983Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20221201100531.xml (deflated 40%)
2022-12-01T10:52:34.4965070Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CompilerTest-20221201100534.xml (deflated 39%)
2022-12-01T10:52:34.4966071Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CompilerTest-20221201100538.xml (deflated 40%)
2022-12-01T10:52:34.4967061Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CompilerTest-20221201100541.xml (deflated 39%)
2022-12-01T10:52:34.4968033Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CompilerTest-20221201100545.xml (deflated 40%)
2022-12-01T10:52:34.4969014Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CompilerTest-20221201100548.xml (deflated 39%)
2022-12-01T10:52:34.4970019Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CompilerTest-20221201100552.xml (deflated 40%)
2022-12-01T10:52:34.4971024Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CompilerTest-20221201100555.xml (deflated 39%)
2022-12-01T10:52:34.4972080Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CompilerTest-20221201100559.xml (deflated 40%)
2022-12-01T10:52:34.4973199Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CompilerTest-20221201100602.xml (deflated 39%)
2022-12-01T10:52:34.4974178Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CompilerTest-20221201100605.xml (deflated 39%)
2022-12-01T10:52:34.4975157Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CompilerTest-20221201100609.xml (deflated 40%)
2022-12-01T10:52:34.4976239Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221201100612.xml (deflated 44%)
2022-12-01T10:52:34.4977408Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221201100616.xml (deflated 45%)
2022-12-01T10:52:34.4978580Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221201100619.xml (deflated 43%)
2022-12-01T10:52:34.4979734Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221201100623.xml (deflated 43%)
2022-12-01T10:52:34.4981055Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221201100626.xml (deflated 45%)
2022-12-01T10:52:34.4982227Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221201100629.xml (deflated 45%)
2022-12-01T10:52:34.4983374Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221201100633.xml (deflated 46%)
2022-12-01T10:52:34.4984548Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221201100636.xml (deflated 46%)
2022-12-01T10:52:34.4985688Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221201100640.xml (deflated 44%)
2022-12-01T10:52:34.4986861Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221201100643.xml (deflated 45%)
2022-12-01T10:52:34.4988052Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221201100646.xml (deflated 45%)
2022-12-01T10:52:34.4989228Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221201100650.xml (deflated 44%)
2022-12-01T10:52:34.4990388Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221201100653.xml (deflated 44%)
2022-12-01T10:52:34.4991542Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221201100657.xml (deflated 44%)
2022-12-01T10:52:34.4992711Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221201100700.xml (deflated 44%)
2022-12-01T10:52:34.4993981Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221201100704.xml (deflated 45%)
2022-12-01T10:52:34.4995138Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221201100707.xml (deflated 45%)
2022-12-01T10:52:34.4996289Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221201100711.xml (deflated 46%)
2022-12-01T10:52:34.4997456Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221201100714.xml (deflated 45%)
2022-12-01T10:52:34.4998630Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221201100717.xml (deflated 50%)
2022-12-01T10:52:34.4999805Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221201100721.xml (deflated 42%)
2022-12-01T10:52:34.5000965Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221201100724.xml (deflated 42%)
2022-12-01T10:52:34.5002305Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221201100728.xml (deflated 42%)
2022-12-01T10:52:34.5003467Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221201100731.xml (deflated 42%)
2022-12-01T10:52:34.5004628Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221201100735.xml (deflated 43%)
2022-12-01T10:52:34.5005773Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221201100738.xml (deflated 42%)
2022-12-01T10:52:34.5006937Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221201100742.xml (deflated 42%)
2022-12-01T10:52:34.5008100Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221201100745.xml (deflated 42%)
2022-12-01T10:52:34.5009266Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221201100749.xml (deflated 41%)
2022-12-01T10:52:34.5010427Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221201100752.xml (deflated 45%)
2022-12-01T10:52:34.5011521Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221201100756.xml (deflated 46%)
2022-12-01T10:52:34.5012711Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221201100759.xml (deflated 41%)
2022-12-01T10:52:34.5013777Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221201100802.xml (deflated 42%)
2022-12-01T10:52:34.5014899Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221201100806.xml (deflated 42%)
2022-12-01T10:52:34.5016020Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221201100809.xml (deflated 42%)
2022-12-01T10:52:34.5017148Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221201100813.xml (deflated 42%)
2022-12-01T10:52:34.5018273Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221201100816.xml (deflated 42%)
2022-12-01T10:52:34.5019547Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-GlooProcessGroupWithDispatchedCollectivesTests-20221201100820.xml (deflated 44%)
2022-12-01T10:52:34.5020931Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221201100823.xml (deflated 40%)
2022-12-01T10:52:34.5022096Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221201100827.xml (deflated 41%)
2022-12-01T10:52:34.5023170Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221201100830.xml (deflated 40%)
2022-12-01T10:52:34.5024173Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221201100834.xml (deflated 40%)
2022-12-01T10:52:34.5025242Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221201100838.xml (deflated 40%)
2022-12-01T10:52:34.5026294Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221201100841.xml (deflated 40%)
2022-12-01T10:52:34.5027344Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221201100845.xml (deflated 40%)
2022-12-01T10:52:34.5028369Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221201100850.xml (deflated 42%)
2022-12-01T10:52:34.5029438Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221201100854.xml (deflated 40%)
2022-12-01T10:52:34.5030614Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221201100857.xml (deflated 41%)
2022-12-01T10:52:34.5031689Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221201100901.xml (deflated 41%)
2022-12-01T10:52:34.5032775Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221201100904.xml (deflated 40%)
2022-12-01T10:52:34.5033836Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221201100908.xml (deflated 40%)
2022-12-01T10:52:34.5034924Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221201100912.xml (deflated 40%)
2022-12-01T10:52:34.5035995Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221201100916.xml (deflated 40%)
2022-12-01T10:52:34.5037066Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221201100919.xml (deflated 40%)
2022-12-01T10:52:34.5038127Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221201100923.xml (deflated 41%)
2022-12-01T10:52:34.5039211Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221201100927.xml (deflated 40%)
2022-12-01T10:52:34.5040298Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221201100931.xml (deflated 40%)
2022-12-01T10:52:34.5041387Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221201100935.xml (deflated 41%)
2022-12-01T10:52:34.5042440Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221201100939.xml (deflated 40%)
2022-12-01T10:52:34.5043510Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221201100942.xml (deflated 40%)
2022-12-01T10:52:34.5044594Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221201100946.xml (deflated 41%)
2022-12-01T10:52:34.5045647Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221201100950.xml (deflated 41%)
2022-12-01T10:52:34.5046712Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221201100953.xml (deflated 40%)
2022-12-01T10:52:34.5047797Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221201100957.xml (deflated 41%)
2022-12-01T10:52:34.5048888Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221201101001.xml (deflated 40%)
2022-12-01T10:52:34.5050041Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221201101005.xml (deflated 40%)
2022-12-01T10:52:34.5051108Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221201101008.xml (deflated 42%)
2022-12-01T10:52:34.5052256Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221201101012.xml (deflated 40%)
2022-12-01T10:52:34.5053345Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221201101016.xml (deflated 40%)
2022-12-01T10:52:34.5054440Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221201101019.xml (deflated 41%)
2022-12-01T10:52:34.5055519Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221201101025.xml (deflated 42%)
2022-12-01T10:52:34.5056603Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221201101028.xml (deflated 40%)
2022-12-01T10:52:34.5057773Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221201101032.xml (deflated 40%)
2022-12-01T10:52:34.5058852Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221201101036.xml (deflated 41%)
2022-12-01T10:52:34.5059930Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221201101039.xml (deflated 40%)
2022-12-01T10:52:34.5061084Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221201101043.xml (deflated 40%)
2022-12-01T10:52:34.5062242Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221201101048.xml (deflated 41%)
2022-12-01T10:52:34.5063340Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221201101052.xml (deflated 40%)
2022-12-01T10:52:34.5064442Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221201101056.xml (deflated 40%)
2022-12-01T10:52:34.5065544Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221201101100.xml (deflated 41%)
2022-12-01T10:52:34.5066603Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221201101103.xml (deflated 41%)
2022-12-01T10:52:34.5067679Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221201101107.xml (deflated 40%)
2022-12-01T10:52:34.5068767Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221201101110.xml (deflated 40%)
2022-12-01T10:52:34.5069831Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221201101116.xml (deflated 42%)
2022-12-01T10:52:34.5070891Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221201101117.xml (deflated 40%)
2022-12-01T10:52:34.5071979Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221201101121.xml (deflated 42%)
2022-12-01T10:52:34.5073066Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221201101122.xml (deflated 42%)
2022-12-01T10:52:34.5074147Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221201101126.xml (deflated 40%)
2022-12-01T10:52:34.5075167Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ReducerTest-20221201101130.xml (deflated 39%)
2022-12-01T10:52:34.5076145Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ReducerTest-20221201101131.xml (deflated 40%)
2022-12-01T10:52:34.5077115Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ReducerTest-20221201101133.xml (deflated 39%)
2022-12-01T10:52:34.5078195Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ReducerTest-20221201101134.xml (deflated 40%)
2022-12-01T10:52:34.5079149Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ReducerTest-20221201101136.xml (deflated 39%)
2022-12-01T10:52:34.5080111Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ReducerTest-20221201101137.xml (deflated 40%)
2022-12-01T10:52:34.5081131Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-RendezvousEnvTest-20221201101139.xml (deflated 40%)
2022-12-01T10:52:34.5082124Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-TimeoutTest-20221201101141.xml (deflated 42%)
2022-12-01T10:52:34.5083145Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_state_dict/TEST-TestFSDPStateDict-20221201101142.xml (deflated 95%)
2022-12-01T10:52:34.5084274Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_optim_state/TEST-TestFSDPOptimState-20221201101315.xml (deflated 94%)
2022-12-01T10:52:34.5085596Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_mixed_precision/TEST-TestFSDPMixedPrecisionSharded-20221201101428.xml (deflated 93%)
2022-12-01T10:52:34.5086902Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_mixed_precision/TEST-TestFSDPMixedPrecisionUnsharded-20221201101428.xml (deflated 68%)
2022-12-01T10:52:34.5088040Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_wrap/TEST-TestAutoWrap-20221201101538.xml (deflated 87%)
2022-12-01T10:52:34.5089060Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_wrap/TEST-TestFSDPWrap-20221201101538.xml (deflated 91%)
2022-12-01T10:52:34.5090153Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_comm_hooks/TEST-TestCommunicationHooks-20221201101612.xml (deflated 92%)
2022-12-01T10:52:34.5091259Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_grad_acc/TEST-TestGradAcc-20221201101635.xml (deflated 94%)
2022-12-01T10:52:34.5092425Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_comm/TEST-TestCommunication-20221201101654.xml (deflated 91%)
2022-12-01T10:52:34.5093595Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_sharded_grad_scaler/TEST-TestShardGradScaler-20221201101707.xml (deflated 68%)
2022-12-01T10:52:34.5094902Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_sharded_grad_scaler/TEST-TestShardedGradScalerParityWithDDP-20221201101707.xml (deflated 86%)
2022-12-01T10:52:34.5096043Z   adding: test/test-reports/python-unittest/distributed.test_c10d_common/TEST-CommTest-20221201101721.xml (deflated 39%)
2022-12-01T10:52:34.5097262Z   adding: test/test-reports/python-unittest/distributed.test_c10d_common/TEST-ComputeBucketAssignmentTest-20221201101724.xml (deflated 42%)
2022-12-01T10:52:34.5098454Z   adding: test/test-reports/python-unittest/distributed.test_c10d_common/TEST-ComputeBucketAssignmentTest-20221201101726.xml (deflated 41%)
2022-12-01T10:52:34.5099640Z   adding: test/test-reports/python-unittest/distributed.test_c10d_common/TEST-ComputeBucketAssignmentTest-20221201101728.xml (deflated 41%)
2022-12-01T10:52:34.5100898Z   adding: test/test-reports/python-unittest/distributed.test_c10d_common/TEST-ComputeBucketAssignmentTest-20221201101730.xml (deflated 42%)
2022-12-01T10:52:34.5102073Z   adding: test/test-reports/python-unittest/distributed.test_c10d_common/TEST-PythonProcessGroupExtensionTest-20221201101731.xml (deflated 42%)
2022-12-01T10:52:34.5103300Z   adding: test/test-reports/python-unittest/distributed.test_c10d_common/TEST-PythonProcessGroupExtensionTest-20221201101735.xml (deflated 41%)
2022-12-01T10:52:34.5104518Z   adding: test/test-reports/python-unittest/distributed.test_c10d_common/TEST-PythonProcessGroupExtensionTest-20221201101739.xml (deflated 41%)
2022-12-01T10:52:34.5105810Z   adding: test/test-reports/python-unittest/distributed.test_c10d_common/TEST-PythonProcessGroupExtensionTest-20221201101742.xml (deflated 42%)
2022-12-01T10:52:34.5106922Z   adding: test/test-reports/python-unittest/distributed.test_c10d_common/TEST-ReduceOpTest-20221201101747.xml (deflated 40%)
2022-12-01T10:52:34.5108011Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_tp_integration/TEST-TestTPFSDPIntegration-20221201101748.xml (deflated 81%)
2022-12-01T10:52:34.5109141Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_clip_grad_norm/TEST-TestCalcuGradNorm-20221201101757.xml (deflated 86%)
2022-12-01T10:52:34.5110272Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_clip_grad_norm/TEST-TestClipGradNorm-20221201101757.xml (deflated 87%)
2022-12-01T10:52:34.5111459Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_distributed_checkpoint/TEST-TestDistributedCheckpoint-20221201101821.xml (deflated 62%)
2022-12-01T10:52:34.5112634Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_uneven/TEST-TestUnevenParamShard-20221201101826.xml (deflated 41%)
2022-12-01T10:52:34.5113869Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_multiple_forward/TEST-TestMultiForward-20221201101829.xml (deflated 42%)
2022-12-01T10:52:34.5115080Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_overlap/TEST-TestForwardOverlapWorldSizeOne-20221201101832.xml (deflated 44%)
2022-12-01T10:52:34.5116344Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_overlap/TEST-TestForwardOverlapWorldSizeTwo-20221201101832.xml (deflated 43%)
2022-12-01T10:52:34.5117555Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_checkpoint_wrapper/TEST-CheckpointWrapperTest-20221201101834.xml (deflated 73%)
2022-12-01T10:52:34.5118624Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_utils/TEST-TestUtils-20221201101836.xml (deflated 68%)
2022-12-01T10:52:34.5119849Z   adding: test/test-reports/python-unittest/distributed.algorithms.ddp_comm_hooks.test_ddp_hooks/TEST-DistributedDataParallelCommHookTest-20221201101837.xml (deflated 82%)
2022-12-01T10:52:34.5121255Z   adding: test/test-reports/python-unittest/distributed._shard.checkpoint.test_file_system_checkpoint/TEST-TestDistributedStateDictSaveLoad-20221201101838.xml (deflated 43%)
2022-12-01T10:52:34.5122619Z   adding: test/test-reports/python-unittest/distributed._shard.checkpoint.test_file_system_checkpoint/TEST-TestDistributedReshardOnLoad-20221201101838.xml (deflated 74%)
2022-12-01T10:52:34.5124116Z   adding: test/test-reports/python-unittest/distributed._shard.checkpoint.test_file_system_checkpoint/TEST-TestDistributedStateDictSaveLoadWithSharedTensor-20221201101838.xml (deflated 45%)
2022-12-01T10:52:34.5125453Z   adding: test/test-reports/python-unittest/distributed._shard.checkpoint.test_planner/TEST-TestSavePlan-20221201101840.xml (deflated 72%)
2022-12-01T10:52:34.5126611Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_param_exec_order_wrap/TEST-TestFSDPExecOrder-20221201101841.xml (deflated 77%)
2022-12-01T10:52:34.5127843Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaDdpComparisonTest-20221201104321.xml (deflated 42%)
2022-12-01T10:52:34.5129152Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaDistAutogradTest-20221201104323.xml (deflated 42%)
2022-12-01T10:52:34.5130476Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaDistAutogradTest-20221201104326.xml (deflated 42%)
2022-12-01T10:52:34.5131873Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaDistAutogradTest-20221201104329.xml (deflated 42%)
2022-12-01T10:52:34.5133245Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaRemoteModuleTest-20221201104333.xml (deflated 43%)
2022-12-01T10:52:34.5134655Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaRemoteModuleTest-20221201104336.xml (deflated 42%)
2022-12-01T10:52:34.5136001Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaRemoteModuleTest-20221201104339.xml (deflated 42%)
2022-12-01T10:52:34.5137335Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaRemoteModuleTest-20221201104343.xml (deflated 43%)
2022-12-01T10:52:34.5138590Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaRpcTest-20221201104346.xml (deflated 41%)
2022-12-01T10:52:34.5139803Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20221201104349.xml (deflated 41%)
2022-12-01T10:52:34.5141283Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20221201104353.xml (deflated 41%)
2022-12-01T10:52:34.5142539Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20221201104356.xml (deflated 41%)
2022-12-01T10:52:34.5143894Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20221201104359.xml (deflated 41%)
2022-12-01T10:52:34.5145154Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20221201104402.xml (deflated 41%)
2022-12-01T10:52:34.5146420Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20221201104404.xml (deflated 41%)
2022-12-01T10:52:34.5147688Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20221201104405.xml (deflated 41%)
2022-12-01T10:52:34.5148938Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20221201104407.xml (deflated 41%)
2022-12-01T10:52:34.5150291Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201104408.xml (deflated 43%)
2022-12-01T10:52:34.5151685Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201104411.xml (deflated 43%)
2022-12-01T10:52:34.5153108Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201104415.xml (deflated 44%)
2022-12-01T10:52:34.5154503Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201104418.xml (deflated 44%)
2022-12-01T10:52:34.5155905Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201104422.xml (deflated 45%)
2022-12-01T10:52:34.5157297Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201104425.xml (deflated 44%)
2022-12-01T10:52:34.5158706Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201104429.xml (deflated 44%)
2022-12-01T10:52:34.5160113Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201104432.xml (deflated 44%)
2022-12-01T10:52:34.5161524Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201104435.xml (deflated 44%)
2022-12-01T10:52:34.5162942Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201104439.xml (deflated 45%)
2022-12-01T10:52:34.5164423Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201104442.xml (deflated 44%)
2022-12-01T10:52:34.5165855Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201104446.xml (deflated 44%)
2022-12-01T10:52:34.5167474Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201104449.xml (deflated 44%)
2022-12-01T10:52:34.5168897Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201104452.xml (deflated 44%)
2022-12-01T10:52:34.5170304Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201104456.xml (deflated 44%)
2022-12-01T10:52:34.5171782Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201104459.xml (deflated 44%)
2022-12-01T10:52:34.5173266Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201104503.xml (deflated 43%)
2022-12-01T10:52:34.5174663Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201104504.xml (deflated 44%)
2022-12-01T10:52:34.5176122Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201104508.xml (deflated 44%)
2022-12-01T10:52:34.5177549Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201104511.xml (deflated 44%)
2022-12-01T10:52:34.5178978Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201104515.xml (deflated 43%)
2022-12-01T10:52:34.5180518Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201104518.xml (deflated 44%)
2022-12-01T10:52:34.5181941Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201104522.xml (deflated 44%)
2022-12-01T10:52:34.5183398Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201104525.xml (deflated 44%)
2022-12-01T10:52:34.5184809Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201104528.xml (deflated 43%)
2022-12-01T10:52:34.5186212Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201104530.xml (deflated 44%)
2022-12-01T10:52:34.5187633Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201104534.xml (deflated 44%)
2022-12-01T10:52:34.5189049Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201104537.xml (deflated 44%)
2022-12-01T10:52:34.5190459Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201104541.xml (deflated 44%)
2022-12-01T10:52:34.5191867Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201104544.xml (deflated 44%)
2022-12-01T10:52:34.5193254Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201104548.xml (deflated 44%)
2022-12-01T10:52:34.5194780Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201104551.xml (deflated 44%)
2022-12-01T10:52:34.5196209Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201104554.xml (deflated 44%)
2022-12-01T10:52:34.5197631Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201104558.xml (deflated 44%)
2022-12-01T10:52:34.5199052Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201104601.xml (deflated 44%)
2022-12-01T10:52:34.5200473Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201104605.xml (deflated 43%)
2022-12-01T10:52:34.5201893Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201104608.xml (deflated 44%)
2022-12-01T10:52:34.5203317Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201104612.xml (deflated 43%)
2022-12-01T10:52:34.5204839Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201104615.xml (deflated 44%)
2022-12-01T10:52:34.5206229Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201104618.xml (deflated 44%)
2022-12-01T10:52:34.5207622Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201104622.xml (deflated 44%)
2022-12-01T10:52:34.5209039Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201104625.xml (deflated 44%)
2022-12-01T10:52:34.5210456Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201104629.xml (deflated 44%)
2022-12-01T10:52:34.5211922Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201104632.xml (deflated 44%)
2022-12-01T10:52:34.5213339Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201104636.xml (deflated 43%)
2022-12-01T10:52:34.5214755Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201104639.xml (deflated 44%)
2022-12-01T10:52:34.5216158Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201104643.xml (deflated 44%)
2022-12-01T10:52:34.5217575Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201104646.xml (deflated 45%)
2022-12-01T10:52:34.5218977Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201104650.xml (deflated 45%)
2022-12-01T10:52:34.5220483Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201104653.xml (deflated 44%)
2022-12-01T10:52:34.5221864Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201104657.xml (deflated 43%)
2022-12-01T10:52:34.5222820Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201104700.xml (deflated 44%)
2022-12-01T10:52:34.5223825Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201104703.xml (deflated 44%)
2022-12-01T10:52:34.5224748Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201104707.xml (deflated 44%)
2022-12-01T10:52:34.5225626Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201104710.xml (deflated 44%)
2022-12-01T10:52:34.5226522Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201104714.xml (deflated 44%)
2022-12-01T10:52:34.5227413Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201104717.xml (deflated 43%)
2022-12-01T10:52:34.5228259Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201104721.xml (deflated 43%)
2022-12-01T10:52:34.5229134Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201104724.xml (deflated 44%)
2022-12-01T10:52:34.5230137Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201104728.xml (deflated 44%)
2022-12-01T10:52:34.5231022Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201104731.xml (deflated 44%)
2022-12-01T10:52:34.5231906Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201104734.xml (deflated 44%)
2022-12-01T10:52:34.5232842Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201104738.xml (deflated 44%)
2022-12-01T10:52:34.5233719Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201104741.xml (deflated 44%)
2022-12-01T10:52:34.5234649Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201104745.xml (deflated 44%)
2022-12-01T10:52:34.5235559Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201104748.xml (deflated 44%)
2022-12-01T10:52:34.5236405Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201104751.xml (deflated 44%)
2022-12-01T10:52:34.5237331Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201104755.xml (deflated 44%)
2022-12-01T10:52:34.5238198Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201104758.xml (deflated 43%)
2022-12-01T10:52:34.5239105Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201104802.xml (deflated 43%)
2022-12-01T10:52:34.5239972Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201104805.xml (deflated 43%)
2022-12-01T10:52:34.5240882Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201104809.xml (deflated 43%)
2022-12-01T10:52:34.5241772Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201104812.xml (deflated 43%)
2022-12-01T10:52:34.5242637Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201104816.xml (deflated 43%)
2022-12-01T10:52:34.5243547Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201104819.xml (deflated 43%)
2022-12-01T10:52:34.5244466Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201104821.xml (deflated 43%)
2022-12-01T10:52:34.5245322Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201104824.xml (deflated 43%)
2022-12-01T10:52:34.5246171Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201104828.xml (deflated 43%)
2022-12-01T10:52:34.5247053Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201104831.xml (deflated 43%)
2022-12-01T10:52:34.5247945Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201104834.xml (deflated 43%)
2022-12-01T10:52:34.5248895Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201104838.xml (deflated 43%)
2022-12-01T10:52:34.5249811Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201104841.xml (deflated 43%)
2022-12-01T10:52:34.5250725Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201104845.xml (deflated 43%)
2022-12-01T10:52:34.5251564Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201104848.xml (deflated 43%)
2022-12-01T10:52:34.5252586Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201104852.xml (deflated 43%)
2022-12-01T10:52:34.5253456Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201104855.xml (deflated 44%)
2022-12-01T10:52:34.5254362Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201104858.xml (deflated 44%)
2022-12-01T10:52:34.5255306Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeCudaDistAutogradTest-20221201104902.xml (deflated 44%)
2022-12-01T10:52:34.5256246Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeCudaDistAutogradTest-20221201104905.xml (deflated 44%)
2022-12-01T10:52:34.5257193Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeCudaDistAutogradTest-20221201104909.xml (deflated 44%)
2022-12-01T10:52:34.5258088Z   adding: test/test-reports/python-unittest/distributed.optim.test_zero_redundancy_optimizer/TEST-TestZeroRedundancyOptimizerDistributed-20221201104913.xml (deflated 91%)
2022-12-01T10:52:34.5259031Z   adding: test/test-reports/python-unittest/distributed.optim.test_zero_redundancy_optimizer/TEST-TestZeroRedundancyOptimizerSingleRank-20221201104913.xml (deflated 73%)
2022-12-01T10:52:34.5259908Z   adding: test/test-reports/python-unittest/distributed.elastic.multiprocessing.api_test/TEST-RunProcResultsTest-20221201104947.xml (deflated 56%)
2022-12-01T10:52:34.5260878Z   adding: test/test-reports/python-unittest/distributed.elastic.multiprocessing.api_test/TEST-StartProcessesListTest-20221201104947.xml (deflated 80%)
2022-12-01T10:52:34.5261710Z   adding: test/test-reports/python-unittest/distributed.elastic.multiprocessing.api_test/TEST-StartProcessesTest-20221201104947.xml (deflated 79%)
2022-12-01T10:52:34.5262521Z   adding: test/test-reports/python-unittest/distributed.elastic.multiprocessing.api_test/TEST-StdTest-20221201104947.xml (deflated 63%)
2022-12-01T10:52:34.5263267Z   adding: test/test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20221201105006.xml (deflated 41%)
2022-12-01T10:52:34.5264026Z   adding: test/test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20221201105010.xml (deflated 41%)
2022-12-01T10:52:34.5264775Z   adding: test/test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20221201105014.xml (deflated 41%)
2022-12-01T10:52:34.5265538Z   adding: test/test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20221201105017.xml (deflated 41%)
2022-12-01T10:52:34.5266279Z   adding: test/test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20221201105021.xml (deflated 41%)
2022-12-01T10:52:34.5267014Z   adding: test/test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20221201105025.xml (deflated 41%)
2022-12-01T10:52:34.5267779Z   adding: test/test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20221201105029.xml (deflated 41%)
2022-12-01T10:52:34.5268596Z   adding: test/test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20221201105032.xml (deflated 41%)
2022-12-01T10:52:34.5269339Z   adding: test/test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20221201105036.xml (deflated 40%)
2022-12-01T10:52:34.5270065Z   adding: test/test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupNCCLWrapperTest-20221201105040.xml (deflated 42%)
2022-12-01T10:52:34.5270824Z   adding: test/test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupNCCLWrapperTest-20221201105041.xml (deflated 41%)
2022-12-01T10:52:34.5271575Z   adding: test/test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupNCCLWrapperTest-20221201105043.xml (deflated 41%)
2022-12-01T10:52:34.5272361Z   adding: test/test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupNCCLWrapperTest-20221201105044.xml (deflated 41%)
2022-12-01T10:52:34.5273094Z   adding: test/test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupNCCLWrapperTest-20221201105046.xml (deflated 41%)
2022-12-01T10:52:34.5273872Z   adding: test/test-reports/python-unittest/distributed.elastic.timer.local_timer_example/TEST-LocalTimerExample-20221201105047.xml (deflated 54%)
2022-12-01T10:52:34.5274660Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardedTensorMetadata-20221201105120.xml (deflated 44%)
2022-12-01T10:52:34.5275533Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestCreateTensorFromParams-20221201105120.xml (deflated 44%)
2022-12-01T10:52:34.5276296Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestLocalTensor-20221201105120.xml (deflated 62%)
2022-12-01T10:52:34.5277051Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestModuleHookApi-20221201105120.xml (deflated 61%)
2022-12-01T10:52:34.5277814Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardMetadata-20221201105120.xml (deflated 60%)
2022-12-01T10:52:34.5278625Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardParameter-20221201105120.xml (deflated 62%)
2022-12-01T10:52:34.5279371Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardTensor-20221201105120.xml (deflated 62%)
2022-12-01T10:52:34.5280179Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardedTensorChunked-20221201105120.xml (deflated 90%)
2022-12-01T10:52:34.5281055Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardedTensorCustomOps-20221201105120.xml (deflated 71%)
2022-12-01T10:52:34.5281940Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardedTensorEnumerable-20221201105120.xml (deflated 88%)
2022-12-01T10:52:34.5282809Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardedTensorFromLocalShards-20221201105120.xml (deflated 86%)
2022-12-01T10:52:34.5283640Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardedTensorFromLocalTensor-20221201105120.xml (deflated 63%)
2022-12-01T10:52:34.5284493Z   adding: test/test-reports/python-unittest/distributed.elastic.utils.distributed_test/TEST-DistributedUtilTest-20221201105124.xml (deflated 71%)
2022-12-01T10:52:34.5285234Z   adding: test/test-reports/python-unittest/distributed.rpc.test_share_memory/TEST-TestRPCPickler-20221201105130.xml (deflated 39%)
2022-12-01T10:52:34.5285989Z   adding: test/test-reports/python-unittest/distributed.elastic.utils.logging_test/TEST-LoggingTest-20221201105133.xml (deflated 55%)
2022-12-01T10:52:34.5286724Z   adding: test/test-reports/python-unittest/distributed.nn.jit.test_instantiator/TEST-TestInstantiator-20221201105134.xml (deflated 64%)
2022-12-01T10:52:34.5287414Z   adding: test/test-reports/python-unittest/distributed.test_data_parallel/TEST-TestDataParallel-20221201105136.xml (deflated 90%)
2022-12-01T10:52:34.5288183Z   adding: test/test-reports/python-unittest/distributed.test_data_parallel/TEST-TestDataParallelDeviceTypeCPU-20221201105136.xml (deflated 91%)
2022-12-01T10:52:34.5288983Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_matrix_ops/TEST-TestShardedTensorMatrixOps-20221201105137.xml (deflated 88%)
2022-12-01T10:52:34.5289792Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_linear/TEST-TestShardedTensorOpsLinear-20221201105138.xml (deflated 71%)
2022-12-01T10:52:34.5290684Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_elementwise_ops/TEST-TestShardedTensorElementWiseOps-20221201105140.xml (deflated 76%)
2022-12-01T10:52:34.5291481Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-ProcessGroupShareTensorTest-20221201105142.xml (deflated 42%)
2022-12-01T10:52:34.5292321Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-ProcessGroupShareTensorTest-20221201105144.xml (deflated 42%)
2022-12-01T10:52:34.5293099Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-ProcessGroupShareTensorTest-20221201105145.xml (deflated 42%)
2022-12-01T10:52:34.5293894Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-ProcessGroupShareTensorTest-20221201105147.xml (deflated 42%)
2022-12-01T10:52:34.5294687Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20221201105148.xml (deflated 44%)
2022-12-01T10:52:34.5295456Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20221201105149.xml (deflated 43%)
2022-12-01T10:52:34.5296229Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20221201105151.xml (deflated 43%)
2022-12-01T10:52:34.5296991Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20221201105152.xml (deflated 44%)
2022-12-01T10:52:34.5297800Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20221201105154.xml (deflated 44%)
2022-12-01T10:52:34.5298591Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20221201105155.xml (deflated 44%)
2022-12-01T10:52:34.5299438Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20221201105157.xml (deflated 44%)
2022-12-01T10:52:34.5300257Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20221201105158.xml (deflated 43%)
2022-12-01T10:52:34.5301200Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20221201105200.xml (deflated 43%)
2022-12-01T10:52:34.5301984Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_chunk/TEST-TestShardedTensorChunkOps-20221201105201.xml (deflated 62%)
2022-12-01T10:52:34.5302878Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.test_megatron_prototype/TEST-TestShardedTensorMegatronLinear-20221201105202.xml (deflated 45%)
2022-12-01T10:52:34.5303712Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_binary_cmp/TEST-TestShardedTensorBinaryOps-20221201105203.xml (deflated 75%)
2022-12-01T10:52:34.5304584Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201101850.xml (deflated 43%)
2022-12-01T10:52:34.5305282Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201101853.xml (deflated 42%)
2022-12-01T10:52:34.5306037Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201101855.xml (deflated 42%)
2022-12-01T10:52:34.5306728Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201101858.xml (deflated 42%)
2022-12-01T10:52:34.5307438Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201101859.xml (deflated 42%)
2022-12-01T10:52:34.5308177Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201101903.xml (deflated 41%)
2022-12-01T10:52:34.5308914Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201101906.xml (deflated 41%)
2022-12-01T10:52:34.5309635Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201101910.xml (deflated 41%)
2022-12-01T10:52:34.5310327Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201101913.xml (deflated 41%)
2022-12-01T10:52:34.5311010Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201101916.xml (deflated 40%)
2022-12-01T10:52:34.5311745Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201101919.xml (deflated 41%)
2022-12-01T10:52:34.5312441Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201101923.xml (deflated 41%)
2022-12-01T10:52:34.5313162Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201101926.xml (deflated 41%)
2022-12-01T10:52:34.5313891Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201101929.xml (deflated 42%)
2022-12-01T10:52:34.5314604Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201101931.xml (deflated 42%)
2022-12-01T10:52:34.5315325Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201101934.xml (deflated 42%)
2022-12-01T10:52:34.5316025Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201101936.xml (deflated 43%)
2022-12-01T10:52:34.5316737Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201101939.xml (deflated 42%)
2022-12-01T10:52:34.5317519Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201101941.xml (deflated 46%)
2022-12-01T10:52:34.5318250Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201101944.xml (deflated 47%)
2022-12-01T10:52:34.5318952Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201101947.xml (deflated 48%)
2022-12-01T10:52:34.5319654Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201101950.xml (deflated 46%)
2022-12-01T10:52:34.5320366Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201101954.xml (deflated 41%)
2022-12-01T10:52:34.5321053Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201101957.xml (deflated 41%)
2022-12-01T10:52:34.5321745Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102000.xml (deflated 41%)
2022-12-01T10:52:34.5322525Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102004.xml (deflated 41%)
2022-12-01T10:52:34.5323244Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102007.xml (deflated 41%)
2022-12-01T10:52:34.5323965Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102011.xml (deflated 41%)
2022-12-01T10:52:34.5324660Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102014.xml (deflated 41%)
2022-12-01T10:52:34.5325331Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102017.xml (deflated 42%)
2022-12-01T10:52:34.5326038Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102019.xml (deflated 42%)
2022-12-01T10:52:34.5326755Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102020.xml (deflated 42%)
2022-12-01T10:52:34.5327460Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102023.xml (deflated 41%)
2022-12-01T10:52:34.5328150Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102027.xml (deflated 43%)
2022-12-01T10:52:34.5328840Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102028.xml (deflated 43%)
2022-12-01T10:52:34.5329519Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102030.xml (deflated 43%)
2022-12-01T10:52:34.5330279Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102031.xml (deflated 43%)
2022-12-01T10:52:34.5330989Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102032.xml (deflated 40%)
2022-12-01T10:52:34.5331701Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102036.xml (deflated 40%)
2022-12-01T10:52:34.5332495Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102039.xml (deflated 44%)
2022-12-01T10:52:34.5333178Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102041.xml (deflated 41%)
2022-12-01T10:52:34.5333914Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102044.xml (deflated 41%)
2022-12-01T10:52:34.5334686Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102047.xml (deflated 41%)
2022-12-01T10:52:34.5335419Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102051.xml (deflated 41%)
2022-12-01T10:52:34.5336149Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102054.xml (deflated 41%)
2022-12-01T10:52:34.5336850Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102058.xml (deflated 41%)
2022-12-01T10:52:34.5337575Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102101.xml (deflated 42%)
2022-12-01T10:52:34.5338316Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102105.xml (deflated 41%)
2022-12-01T10:52:34.5339030Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102108.xml (deflated 41%)
2022-12-01T10:52:34.5339737Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102111.xml (deflated 41%)
2022-12-01T10:52:34.5340669Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102115.xml (deflated 41%)
2022-12-01T10:52:34.5341270Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102118.xml (deflated 41%)
2022-12-01T10:52:34.5342018Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102121.xml (deflated 41%)
2022-12-01T10:52:34.5342714Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102125.xml (deflated 41%)
2022-12-01T10:52:34.5343405Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102128.xml (deflated 41%)
2022-12-01T10:52:34.5344081Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102131.xml (deflated 41%)
2022-12-01T10:52:34.5345131Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102135.xml (deflated 41%)
2022-12-01T10:52:34.5345838Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102138.xml (deflated 41%)
2022-12-01T10:52:34.5346545Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102141.xml (deflated 41%)
2022-12-01T10:52:34.5347217Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102145.xml (deflated 41%)
2022-12-01T10:52:34.5347928Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102148.xml (deflated 41%)
2022-12-01T10:52:34.5348649Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102152.xml (deflated 41%)
2022-12-01T10:52:34.5349331Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102155.xml (deflated 41%)
2022-12-01T10:52:34.5350052Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102159.xml (deflated 41%)
2022-12-01T10:52:34.5350788Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102202.xml (deflated 42%)
2022-12-01T10:52:34.5351486Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102205.xml (deflated 42%)
2022-12-01T10:52:34.5352161Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102208.xml (deflated 41%)
2022-12-01T10:52:34.5353011Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102212.xml (deflated 43%)
2022-12-01T10:52:34.5353735Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102215.xml (deflated 41%)
2022-12-01T10:52:34.5354477Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102218.xml (deflated 42%)
2022-12-01T10:52:34.5355164Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102222.xml (deflated 41%)
2022-12-01T10:52:34.5355893Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102225.xml (deflated 42%)
2022-12-01T10:52:34.5356609Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102228.xml (deflated 42%)
2022-12-01T10:52:34.5357310Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102232.xml (deflated 42%)
2022-12-01T10:52:34.5358101Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102235.xml (deflated 44%)
2022-12-01T10:52:34.5358812Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102236.xml (deflated 43%)
2022-12-01T10:52:34.5359561Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102238.xml (deflated 43%)
2022-12-01T10:52:34.5360279Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102239.xml (deflated 42%)
2022-12-01T10:52:34.5360973Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102240.xml (deflated 43%)
2022-12-01T10:52:34.5361670Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102242.xml (deflated 42%)
2022-12-01T10:52:34.5362380Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102243.xml (deflated 43%)
2022-12-01T10:52:34.5363068Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102244.xml (deflated 43%)
2022-12-01T10:52:34.5363773Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102246.xml (deflated 43%)
2022-12-01T10:52:34.5364472Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102247.xml (deflated 43%)
2022-12-01T10:52:34.5365145Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102248.xml (deflated 43%)
2022-12-01T10:52:34.5365870Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102250.xml (deflated 43%)
2022-12-01T10:52:34.5366584Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102251.xml (deflated 43%)
2022-12-01T10:52:34.5367298Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102252.xml (deflated 43%)
2022-12-01T10:52:34.5367893Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102254.xml (deflated 43%)
2022-12-01T10:52:34.5368482Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102255.xml (deflated 43%)
2022-12-01T10:52:34.5369059Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102256.xml (deflated 43%)
2022-12-01T10:52:34.5369645Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102258.xml (deflated 43%)
2022-12-01T10:52:34.5370286Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102259.xml (deflated 43%)
2022-12-01T10:52:34.5370884Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102300.xml (deflated 43%)
2022-12-01T10:52:34.5371468Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102302.xml (deflated 43%)
2022-12-01T10:52:34.5372180Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102303.xml (deflated 43%)
2022-12-01T10:52:34.5372767Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102305.xml (deflated 43%)
2022-12-01T10:52:34.5373352Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102306.xml (deflated 43%)
2022-12-01T10:52:34.5373932Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102307.xml (deflated 42%)
2022-12-01T10:52:34.5375080Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102310.xml (deflated 42%)
2022-12-01T10:52:34.5376097Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102312.xml (deflated 42%)
2022-12-01T10:52:34.5376695Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102313.xml (deflated 41%)
2022-12-01T10:52:34.5377277Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102318.xml (deflated 42%)
2022-12-01T10:52:34.5377869Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102321.xml (deflated 41%)
2022-12-01T10:52:34.5378462Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102326.xml (deflated 41%)
2022-12-01T10:52:34.5379053Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102329.xml (deflated 41%)
2022-12-01T10:52:34.5379629Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102333.xml (deflated 42%)
2022-12-01T10:52:34.5380228Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102336.xml (deflated 40%)
2022-12-01T10:52:34.5380924Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102341.xml (deflated 42%)
2022-12-01T10:52:34.5381513Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102342.xml (deflated 41%)
2022-12-01T10:52:34.5382096Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102351.xml (deflated 41%)
2022-12-01T10:52:34.5382696Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102354.xml (deflated 40%)
2022-12-01T10:52:34.5383290Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102358.xml (deflated 42%)
2022-12-01T10:52:34.5383879Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102359.xml (deflated 42%)
2022-12-01T10:52:34.5384456Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102400.xml (deflated 42%)
2022-12-01T10:52:34.5385050Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102402.xml (deflated 41%)
2022-12-01T10:52:34.5385725Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102403.xml (deflated 42%)
2022-12-01T10:52:34.5386322Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102404.xml (deflated 42%)
2022-12-01T10:52:34.5386900Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102406.xml (deflated 42%)
2022-12-01T10:52:34.5387487Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102407.xml (deflated 42%)
2022-12-01T10:52:34.5388079Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102408.xml (deflated 41%)
2022-12-01T10:52:34.5388671Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102412.xml (deflated 42%)
2022-12-01T10:52:34.5389250Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102413.xml (deflated 40%)
2022-12-01T10:52:34.5389848Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102417.xml (deflated 41%)
2022-12-01T10:52:34.5390510Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102420.xml (deflated 42%)
2022-12-01T10:52:34.5391110Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102423.xml (deflated 42%)
2022-12-01T10:52:34.5391700Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102425.xml (deflated 41%)
2022-12-01T10:52:34.5392277Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102426.xml (deflated 41%)
2022-12-01T10:52:34.5392865Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102428.xml (deflated 42%)
2022-12-01T10:52:34.5393472Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102431.xml (deflated 41%)
2022-12-01T10:52:34.5394067Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102434.xml (deflated 42%)
2022-12-01T10:52:34.5394642Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102436.xml (deflated 42%)
2022-12-01T10:52:34.5395232Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102437.xml (deflated 41%)
2022-12-01T10:52:34.5395818Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102439.xml (deflated 41%)
2022-12-01T10:52:34.5396406Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102442.xml (deflated 41%)
2022-12-01T10:52:34.5396984Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102445.xml (deflated 41%)
2022-12-01T10:52:34.5397574Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102447.xml (deflated 41%)
2022-12-01T10:52:34.5398165Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102448.xml (deflated 41%)
2022-12-01T10:52:34.5398754Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102451.xml (deflated 42%)
2022-12-01T10:52:34.5399333Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102453.xml (deflated 41%)
2022-12-01T10:52:34.5399922Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102454.xml (deflated 42%)
2022-12-01T10:52:34.5400553Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102456.xml (deflated 42%)
2022-12-01T10:52:34.5401154Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102457.xml (deflated 41%)
2022-12-01T10:52:34.5401735Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102501.xml (deflated 41%)
2022-12-01T10:52:34.5402363Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102504.xml (deflated 42%)
2022-12-01T10:52:34.5402954Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102506.xml (deflated 41%)
2022-12-01T10:52:34.5403580Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102509.xml (deflated 42%)
2022-12-01T10:52:34.5404162Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102512.xml (deflated 42%)
2022-12-01T10:52:34.5404790Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102515.xml (deflated 42%)
2022-12-01T10:52:34.5405375Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102519.xml (deflated 42%)
2022-12-01T10:52:34.5405960Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102522.xml (deflated 42%)
2022-12-01T10:52:34.5406542Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102525.xml (deflated 42%)
2022-12-01T10:52:34.5407128Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102528.xml (deflated 42%)
2022-12-01T10:52:34.5407714Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102532.xml (deflated 41%)
2022-12-01T10:52:34.5408307Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102535.xml (deflated 42%)
2022-12-01T10:52:34.5408917Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102538.xml (deflated 42%)
2022-12-01T10:52:34.5409504Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102542.xml (deflated 42%)
2022-12-01T10:52:34.5410090Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102545.xml (deflated 42%)
2022-12-01T10:52:34.5410680Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102548.xml (deflated 41%)
2022-12-01T10:52:34.5411257Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102550.xml (deflated 42%)
2022-12-01T10:52:34.5411944Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102553.xml (deflated 41%)
2022-12-01T10:52:34.5412541Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102556.xml (deflated 41%)
2022-12-01T10:52:34.5413133Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102600.xml (deflated 42%)
2022-12-01T10:52:34.5413708Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102603.xml (deflated 41%)
2022-12-01T10:52:34.5414308Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102604.xml (deflated 41%)
2022-12-01T10:52:34.5414896Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102606.xml (deflated 42%)
2022-12-01T10:52:34.5415545Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102607.xml (deflated 42%)
2022-12-01T10:52:34.5416126Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102608.xml (deflated 42%)
2022-12-01T10:52:34.5416718Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102610.xml (deflated 42%)
2022-12-01T10:52:34.5417304Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102613.xml (deflated 42%)
2022-12-01T10:52:34.5417893Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102615.xml (deflated 42%)
2022-12-01T10:52:34.5418470Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102616.xml (deflated 42%)
2022-12-01T10:52:34.5419060Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102617.xml (deflated 42%)
2022-12-01T10:52:34.5419700Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102621.xml (deflated 42%)
2022-12-01T10:52:34.5420292Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102622.xml (deflated 42%)
2022-12-01T10:52:34.5420990Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102624.xml (deflated 42%)
2022-12-01T10:52:34.5421577Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102625.xml (deflated 42%)
2022-12-01T10:52:34.5422162Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102629.xml (deflated 42%)
2022-12-01T10:52:34.5422751Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102632.xml (deflated 42%)
2022-12-01T10:52:34.5423331Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102635.xml (deflated 42%)
2022-12-01T10:52:34.5423925Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102637.xml (deflated 42%)
2022-12-01T10:52:34.5424511Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102639.xml (deflated 42%)
2022-12-01T10:52:34.5425095Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102640.xml (deflated 41%)
2022-12-01T10:52:34.5425676Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102642.xml (deflated 41%)
2022-12-01T10:52:34.5426270Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102645.xml (deflated 41%)
2022-12-01T10:52:34.5426857Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102648.xml (deflated 42%)
2022-12-01T10:52:34.5427447Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102650.xml (deflated 41%)
2022-12-01T10:52:34.5428037Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102651.xml (deflated 41%)
2022-12-01T10:52:34.5428611Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102655.xml (deflated 41%)
2022-12-01T10:52:34.5429200Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102658.xml (deflated 41%)
2022-12-01T10:52:34.5429853Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102701.xml (deflated 43%)
2022-12-01T10:52:34.5430565Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102702.xml (deflated 41%)
2022-12-01T10:52:34.5431146Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102706.xml (deflated 41%)
2022-12-01T10:52:34.5431740Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102709.xml (deflated 41%)
2022-12-01T10:52:34.5432327Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102713.xml (deflated 42%)
2022-12-01T10:52:34.5432918Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102714.xml (deflated 41%)
2022-12-01T10:52:34.5433502Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102718.xml (deflated 40%)
2022-12-01T10:52:34.5434095Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102721.xml (deflated 41%)
2022-12-01T10:52:34.5434739Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102724.xml (deflated 40%)
2022-12-01T10:52:34.5435329Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102728.xml (deflated 41%)
2022-12-01T10:52:34.5435906Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102731.xml (deflated 42%)
2022-12-01T10:52:34.5436494Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102733.xml (deflated 41%)
2022-12-01T10:52:34.5437086Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102736.xml (deflated 41%)
2022-12-01T10:52:34.5437677Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102739.xml (deflated 41%)
2022-12-01T10:52:34.5438259Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102743.xml (deflated 41%)
2022-12-01T10:52:34.5438848Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102746.xml (deflated 42%)
2022-12-01T10:52:34.5439438Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102747.xml (deflated 41%)
2022-12-01T10:52:34.5440026Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102749.xml (deflated 41%)
2022-12-01T10:52:34.5440600Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102754.xml (deflated 41%)
2022-12-01T10:52:34.5441191Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102800.xml (deflated 41%)
2022-12-01T10:52:34.5441777Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102803.xml (deflated 41%)
2022-12-01T10:52:34.5442364Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102807.xml (deflated 41%)
2022-12-01T10:52:34.5442941Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102810.xml (deflated 43%)
2022-12-01T10:52:34.5443525Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102811.xml (deflated 42%)
2022-12-01T10:52:34.5444111Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102813.xml (deflated 42%)
2022-12-01T10:52:34.5444745Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102814.xml (deflated 42%)
2022-12-01T10:52:34.5445330Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102815.xml (deflated 43%)
2022-12-01T10:52:34.5445915Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102817.xml (deflated 43%)
2022-12-01T10:52:34.5446547Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102818.xml (deflated 42%)
2022-12-01T10:52:34.5447133Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102819.xml (deflated 42%)
2022-12-01T10:52:34.5447706Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102821.xml (deflated 42%)
2022-12-01T10:52:34.5448297Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102824.xml (deflated 42%)
2022-12-01T10:52:34.5448947Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102827.xml (deflated 42%)
2022-12-01T10:52:34.5449530Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102829.xml (deflated 43%)
2022-12-01T10:52:34.5450105Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102830.xml (deflated 42%)
2022-12-01T10:52:34.5450698Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102832.xml (deflated 42%)
2022-12-01T10:52:34.5451282Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102835.xml (deflated 42%)
2022-12-01T10:52:34.5452002Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102838.xml (deflated 42%)
2022-12-01T10:52:34.5452581Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102841.xml (deflated 42%)
2022-12-01T10:52:34.5453177Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102843.xml (deflated 42%)
2022-12-01T10:52:34.5453762Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102845.xml (deflated 41%)
2022-12-01T10:52:34.5454350Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102848.xml (deflated 40%)
2022-12-01T10:52:34.5454921Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102851.xml (deflated 42%)
2022-12-01T10:52:34.5455508Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102853.xml (deflated 41%)
2022-12-01T10:52:34.5456099Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102856.xml (deflated 41%)
2022-12-01T10:52:34.5456689Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102900.xml (deflated 41%)
2022-12-01T10:52:34.5457261Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102903.xml (deflated 41%)
2022-12-01T10:52:34.5457845Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102906.xml (deflated 41%)
2022-12-01T10:52:34.5458432Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102910.xml (deflated 41%)
2022-12-01T10:52:34.5459063Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102913.xml (deflated 41%)
2022-12-01T10:52:34.5459686Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102917.xml (deflated 41%)
2022-12-01T10:52:34.5460280Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102920.xml (deflated 41%)
2022-12-01T10:52:34.5460971Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102924.xml (deflated 41%)
2022-12-01T10:52:34.5461566Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102927.xml (deflated 43%)
2022-12-01T10:52:34.5462154Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102928.xml (deflated 41%)
2022-12-01T10:52:34.5462732Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102932.xml (deflated 44%)
2022-12-01T10:52:34.5463323Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102933.xml (deflated 43%)
2022-12-01T10:52:34.5463972Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102934.xml (deflated 41%)
2022-12-01T10:52:34.5464562Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102938.xml (deflated 43%)
2022-12-01T10:52:34.5465135Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102939.xml (deflated 43%)
2022-12-01T10:52:34.5465720Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102940.xml (deflated 41%)
2022-12-01T10:52:34.5466307Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102944.xml (deflated 41%)
2022-12-01T10:52:34.5466896Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102947.xml (deflated 41%)
2022-12-01T10:52:34.5467476Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102950.xml (deflated 41%)
2022-12-01T10:52:34.5468065Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102953.xml (deflated 42%)
2022-12-01T10:52:34.5468650Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102955.xml (deflated 41%)
2022-12-01T10:52:34.5469239Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102956.xml (deflated 41%)
2022-12-01T10:52:34.5469815Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103000.xml (deflated 41%)
2022-12-01T10:52:34.5470400Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103003.xml (deflated 41%)
2022-12-01T10:52:34.5470989Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103006.xml (deflated 41%)
2022-12-01T10:52:34.5471572Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103009.xml (deflated 41%)
2022-12-01T10:52:34.5472147Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103013.xml (deflated 41%)
2022-12-01T10:52:34.5472730Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103016.xml (deflated 41%)
2022-12-01T10:52:34.5473318Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103020.xml (deflated 41%)
2022-12-01T10:52:34.5473952Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103023.xml (deflated 42%)
2022-12-01T10:52:34.5474531Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103024.xml (deflated 42%)
2022-12-01T10:52:34.5475118Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103025.xml (deflated 42%)
2022-12-01T10:52:34.5475702Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103027.xml (deflated 41%)
2022-12-01T10:52:34.5476288Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103030.xml (deflated 41%)
2022-12-01T10:52:34.5476869Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103033.xml (deflated 41%)
2022-12-01T10:52:34.5477453Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103037.xml (deflated 41%)
2022-12-01T10:52:34.5478040Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103040.xml (deflated 41%)
2022-12-01T10:52:34.5478672Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103043.xml (deflated 42%)
2022-12-01T10:52:34.5479245Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103047.xml (deflated 42%)
2022-12-01T10:52:34.5479829Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103050.xml (deflated 41%)
2022-12-01T10:52:34.5480414Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103053.xml (deflated 42%)
2022-12-01T10:52:34.5480996Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103056.xml (deflated 42%)
2022-12-01T10:52:34.5481574Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103100.xml (deflated 41%)
2022-12-01T10:52:34.5482168Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103101.xml (deflated 41%)
2022-12-01T10:52:34.5482752Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103104.xml (deflated 43%)
2022-12-01T10:52:34.5483337Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103107.xml (deflated 42%)
2022-12-01T10:52:34.5483910Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103108.xml (deflated 42%)
2022-12-01T10:52:34.5484494Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103112.xml (deflated 43%)
2022-12-01T10:52:34.5485085Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103113.xml (deflated 42%)
2022-12-01T10:52:34.5485676Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103117.xml (deflated 41%)
2022-12-01T10:52:34.5486249Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103120.xml (deflated 42%)
2022-12-01T10:52:34.5486838Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103123.xml (deflated 41%)
2022-12-01T10:52:34.5487423Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103127.xml (deflated 42%)
2022-12-01T10:52:34.5488008Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103130.xml (deflated 41%)
2022-12-01T10:52:34.5488622Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103133.xml (deflated 41%)
2022-12-01T10:52:34.5489219Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103136.xml (deflated 41%)
2022-12-01T10:52:34.5489859Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103140.xml (deflated 41%)
2022-12-01T10:52:34.5490443Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103143.xml (deflated 42%)
2022-12-01T10:52:34.5491018Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103145.xml (deflated 42%)
2022-12-01T10:52:34.5491606Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103148.xml (deflated 42%)
2022-12-01T10:52:34.5492312Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103150.xml (deflated 43%)
2022-12-01T10:52:34.5492942Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103153.xml (deflated 42%)
2022-12-01T10:52:34.5493516Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103154.xml (deflated 46%)
2022-12-01T10:52:34.5494107Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103158.xml (deflated 47%)
2022-12-01T10:52:34.5494694Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103201.xml (deflated 48%)
2022-12-01T10:52:34.5495282Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103204.xml (deflated 46%)
2022-12-01T10:52:34.5495858Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103207.xml (deflated 41%)
2022-12-01T10:52:34.5496443Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103211.xml (deflated 41%)
2022-12-01T10:52:34.5497031Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103214.xml (deflated 41%)
2022-12-01T10:52:34.5497616Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103217.xml (deflated 41%)
2022-12-01T10:52:34.5498202Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103221.xml (deflated 41%)
2022-12-01T10:52:34.5498773Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103224.xml (deflated 41%)
2022-12-01T10:52:34.5499358Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103227.xml (deflated 41%)
2022-12-01T10:52:34.5499945Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103231.xml (deflated 43%)
2022-12-01T10:52:34.5500674Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103232.xml (deflated 42%)
2022-12-01T10:52:34.5501251Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103233.xml (deflated 41%)
2022-12-01T10:52:34.5501838Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103237.xml (deflated 41%)
2022-12-01T10:52:34.5502423Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103240.xml (deflated 43%)
2022-12-01T10:52:34.5503018Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103242.xml (deflated 43%)
2022-12-01T10:52:34.5503658Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103243.xml (deflated 43%)
2022-12-01T10:52:34.5504255Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103244.xml (deflated 43%)
2022-12-01T10:52:34.5504841Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103246.xml (deflated 40%)
2022-12-01T10:52:34.5505426Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103249.xml (deflated 40%)
2022-12-01T10:52:34.5506000Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103253.xml (deflated 44%)
2022-12-01T10:52:34.5506582Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103254.xml (deflated 41%)
2022-12-01T10:52:34.5507167Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103257.xml (deflated 41%)
2022-12-01T10:52:34.5507810Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103301.xml (deflated 41%)
2022-12-01T10:52:34.5508391Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103304.xml (deflated 41%)
2022-12-01T10:52:34.5508980Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103308.xml (deflated 41%)
2022-12-01T10:52:34.5509564Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103311.xml (deflated 41%)
2022-12-01T10:52:34.5510154Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103314.xml (deflated 41%)
2022-12-01T10:52:34.5510736Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103318.xml (deflated 41%)
2022-12-01T10:52:34.5511328Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103321.xml (deflated 41%)
2022-12-01T10:52:34.5511913Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103324.xml (deflated 41%)
2022-12-01T10:52:34.5512499Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103328.xml (deflated 41%)
2022-12-01T10:52:34.5513073Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103331.xml (deflated 41%)
2022-12-01T10:52:34.5513661Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103335.xml (deflated 41%)
2022-12-01T10:52:34.5514251Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103338.xml (deflated 41%)
2022-12-01T10:52:34.5514840Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103341.xml (deflated 40%)
2022-12-01T10:52:34.5515412Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103345.xml (deflated 40%)
2022-12-01T10:52:34.5515999Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103348.xml (deflated 41%)
2022-12-01T10:52:34.5516580Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103351.xml (deflated 41%)
2022-12-01T10:52:34.5517168Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103355.xml (deflated 40%)
2022-12-01T10:52:34.5517783Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103358.xml (deflated 40%)
2022-12-01T10:52:34.5518368Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103402.xml (deflated 41%)
2022-12-01T10:52:34.5518955Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103405.xml (deflated 41%)
2022-12-01T10:52:34.5519542Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103409.xml (deflated 41%)
2022-12-01T10:52:34.5520118Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103412.xml (deflated 41%)
2022-12-01T10:52:34.5520703Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103416.xml (deflated 42%)
2022-12-01T10:52:34.5521286Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103419.xml (deflated 41%)
2022-12-01T10:52:34.5521871Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103422.xml (deflated 41%)
2022-12-01T10:52:34.5522484Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103425.xml (deflated 42%)
2022-12-01T10:52:34.5523067Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103429.xml (deflated 41%)
2022-12-01T10:52:34.5523654Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103432.xml (deflated 41%)
2022-12-01T10:52:34.5524238Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103435.xml (deflated 41%)
2022-12-01T10:52:34.5524818Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103439.xml (deflated 42%)
2022-12-01T10:52:34.5525407Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103442.xml (deflated 42%)
2022-12-01T10:52:34.5525995Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103445.xml (deflated 41%)
2022-12-01T10:52:34.5526579Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103448.xml (deflated 43%)
2022-12-01T10:52:34.5527155Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103450.xml (deflated 43%)
2022-12-01T10:52:34.5527739Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103451.xml (deflated 43%)
2022-12-01T10:52:34.5528325Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103452.xml (deflated 42%)
2022-12-01T10:52:34.5528914Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103454.xml (deflated 43%)
2022-12-01T10:52:34.5529486Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103455.xml (deflated 42%)
2022-12-01T10:52:34.5530075Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103456.xml (deflated 43%)
2022-12-01T10:52:34.5530656Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103458.xml (deflated 42%)
2022-12-01T10:52:34.5531246Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103459.xml (deflated 43%)
2022-12-01T10:52:34.5531905Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103500.xml (deflated 43%)
2022-12-01T10:52:34.5532543Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103502.xml (deflated 43%)
2022-12-01T10:52:34.5533130Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103503.xml (deflated 43%)
2022-12-01T10:52:34.5533712Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103504.xml (deflated 43%)
2022-12-01T10:52:34.5534298Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103506.xml (deflated 43%)
2022-12-01T10:52:34.5534877Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103507.xml (deflated 43%)
2022-12-01T10:52:34.5535456Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103508.xml (deflated 43%)
2022-12-01T10:52:34.5536041Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103510.xml (deflated 43%)
2022-12-01T10:52:34.5536668Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103511.xml (deflated 43%)
2022-12-01T10:52:34.5537246Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103512.xml (deflated 43%)
2022-12-01T10:52:34.5537829Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103514.xml (deflated 43%)
2022-12-01T10:52:34.5538417Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103515.xml (deflated 43%)
2022-12-01T10:52:34.5539002Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103516.xml (deflated 43%)
2022-12-01T10:52:34.5539581Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103518.xml (deflated 43%)
2022-12-01T10:52:34.5540170Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103519.xml (deflated 43%)
2022-12-01T10:52:34.5540864Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103520.xml (deflated 42%)
2022-12-01T10:52:34.5541449Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103524.xml (deflated 42%)
2022-12-01T10:52:34.5542023Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103525.xml (deflated 42%)
2022-12-01T10:52:34.5542611Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103526.xml (deflated 41%)
2022-12-01T10:52:34.5543196Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103531.xml (deflated 42%)
2022-12-01T10:52:34.5543788Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103534.xml (deflated 40%)
2022-12-01T10:52:34.5544366Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103539.xml (deflated 41%)
2022-12-01T10:52:34.5544951Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103542.xml (deflated 41%)
2022-12-01T10:52:34.5545540Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103546.xml (deflated 42%)
2022-12-01T10:52:34.5546128Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103550.xml (deflated 41%)
2022-12-01T10:52:34.5546702Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103554.xml (deflated 41%)
2022-12-01T10:52:34.5547354Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103559.xml (deflated 41%)
2022-12-01T10:52:34.5547951Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103607.xml (deflated 41%)
2022-12-01T10:52:34.5548540Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103610.xml (deflated 40%)
2022-12-01T10:52:34.5549115Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103614.xml (deflated 42%)
2022-12-01T10:52:34.5549699Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103615.xml (deflated 42%)
2022-12-01T10:52:34.5550281Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103616.xml (deflated 41%)
2022-12-01T10:52:34.5550866Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103618.xml (deflated 41%)
2022-12-01T10:52:34.5551504Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103619.xml (deflated 41%)
2022-12-01T10:52:34.5552092Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103620.xml (deflated 42%)
2022-12-01T10:52:34.5552684Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103622.xml (deflated 41%)
2022-12-01T10:52:34.5553269Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103623.xml (deflated 42%)
2022-12-01T10:52:34.5553846Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103624.xml (deflated 41%)
2022-12-01T10:52:34.5554433Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103628.xml (deflated 42%)
2022-12-01T10:52:34.5555017Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103629.xml (deflated 41%)
2022-12-01T10:52:34.5555600Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103633.xml (deflated 41%)
2022-12-01T10:52:34.5556174Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103636.xml (deflated 41%)
2022-12-01T10:52:34.5556758Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103639.xml (deflated 42%)
2022-12-01T10:52:34.5557343Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103641.xml (deflated 41%)
2022-12-01T10:52:34.5557926Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103642.xml (deflated 41%)
2022-12-01T10:52:34.5558501Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103644.xml (deflated 42%)
2022-12-01T10:52:34.5559091Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103647.xml (deflated 41%)
2022-12-01T10:52:34.5559679Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103650.xml (deflated 42%)
2022-12-01T10:52:34.5560263Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103652.xml (deflated 42%)
2022-12-01T10:52:34.5560835Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103654.xml (deflated 41%)
2022-12-01T10:52:34.5561467Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103655.xml (deflated 41%)
2022-12-01T10:52:34.5562054Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103658.xml (deflated 41%)
2022-12-01T10:52:34.5562645Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103701.xml (deflated 41%)
2022-12-01T10:52:34.5563217Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103703.xml (deflated 41%)
2022-12-01T10:52:34.5563804Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103704.xml (deflated 41%)
2022-12-01T10:52:34.5564392Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103707.xml (deflated 42%)
2022-12-01T10:52:34.5564976Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103709.xml (deflated 42%)
2022-12-01T10:52:34.5565548Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103710.xml (deflated 42%)
2022-12-01T10:52:34.5566177Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103712.xml (deflated 42%)
2022-12-01T10:52:34.5566764Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103713.xml (deflated 41%)
2022-12-01T10:52:34.5567347Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103717.xml (deflated 41%)
2022-12-01T10:52:34.5567918Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103720.xml (deflated 42%)
2022-12-01T10:52:34.5568504Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103722.xml (deflated 41%)
2022-12-01T10:52:34.5569088Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103725.xml (deflated 42%)
2022-12-01T10:52:34.5569677Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103728.xml (deflated 42%)
2022-12-01T10:52:34.5570276Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103731.xml (deflated 42%)
2022-12-01T10:52:34.5570853Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103735.xml (deflated 42%)
2022-12-01T10:52:34.5571434Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103738.xml (deflated 42%)
2022-12-01T10:52:34.5572110Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103741.xml (deflated 41%)
2022-12-01T10:52:34.5572699Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103744.xml (deflated 42%)
2022-12-01T10:52:34.5573276Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103748.xml (deflated 41%)
2022-12-01T10:52:34.5573865Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103751.xml (deflated 42%)
2022-12-01T10:52:34.5574449Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103754.xml (deflated 42%)
2022-12-01T10:52:34.5575035Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103757.xml (deflated 42%)
2022-12-01T10:52:34.5575610Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103801.xml (deflated 42%)
2022-12-01T10:52:34.5576240Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103804.xml (deflated 41%)
2022-12-01T10:52:34.5576828Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103805.xml (deflated 42%)
2022-12-01T10:52:34.5577414Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103809.xml (deflated 41%)
2022-12-01T10:52:34.5577993Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103812.xml (deflated 40%)
2022-12-01T10:52:34.5578578Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103815.xml (deflated 42%)
2022-12-01T10:52:34.5579167Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103819.xml (deflated 41%)
2022-12-01T10:52:34.5579751Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103820.xml (deflated 41%)
2022-12-01T10:52:34.5580465Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103821.xml (deflated 42%)
2022-12-01T10:52:34.5581058Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103823.xml (deflated 42%)
2022-12-01T10:52:34.5581640Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103824.xml (deflated 42%)
2022-12-01T10:52:34.5582227Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103825.xml (deflated 42%)
2022-12-01T10:52:34.5582800Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103828.xml (deflated 42%)
2022-12-01T10:52:34.5583387Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103830.xml (deflated 42%)
2022-12-01T10:52:34.5583973Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103831.xml (deflated 42%)
2022-12-01T10:52:34.5584559Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103833.xml (deflated 42%)
2022-12-01T10:52:34.5585137Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103836.xml (deflated 42%)
2022-12-01T10:52:34.5585721Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103838.xml (deflated 41%)
2022-12-01T10:52:34.5586302Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103839.xml (deflated 42%)
2022-12-01T10:52:34.5586888Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103841.xml (deflated 41%)
2022-12-01T10:52:34.5587464Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103844.xml (deflated 42%)
2022-12-01T10:52:34.5588046Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103847.xml (deflated 41%)
2022-12-01T10:52:34.5588632Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103851.xml (deflated 42%)
2022-12-01T10:52:34.5589215Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103852.xml (deflated 42%)
2022-12-01T10:52:34.5589788Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103854.xml (deflated 42%)
2022-12-01T10:52:34.5590369Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103855.xml (deflated 41%)
2022-12-01T10:52:34.5591014Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103857.xml (deflated 41%)
2022-12-01T10:52:34.5591602Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103900.xml (deflated 41%)
2022-12-01T10:52:34.5592178Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103904.xml (deflated 42%)
2022-12-01T10:52:34.5592761Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103906.xml (deflated 41%)
2022-12-01T10:52:34.5593348Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103907.xml (deflated 40%)
2022-12-01T10:52:34.5593930Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103910.xml (deflated 41%)
2022-12-01T10:52:34.5594504Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103914.xml (deflated 41%)
2022-12-01T10:52:34.5595135Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103917.xml (deflated 43%)
2022-12-01T10:52:34.5595718Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103918.xml (deflated 41%)
2022-12-01T10:52:34.5596303Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103922.xml (deflated 41%)
2022-12-01T10:52:34.5596875Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103925.xml (deflated 40%)
2022-12-01T10:52:34.5597463Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103929.xml (deflated 42%)
2022-12-01T10:52:34.5598051Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103930.xml (deflated 41%)
2022-12-01T10:52:34.5598638Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103934.xml (deflated 40%)
2022-12-01T10:52:34.5599209Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103937.xml (deflated 41%)
2022-12-01T10:52:34.5599794Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103940.xml (deflated 40%)
2022-12-01T10:52:34.5600378Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103944.xml (deflated 40%)
2022-12-01T10:52:34.5600963Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103947.xml (deflated 41%)
2022-12-01T10:52:34.5601534Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103949.xml (deflated 41%)
2022-12-01T10:52:34.5602125Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103952.xml (deflated 41%)
2022-12-01T10:52:34.5602711Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103955.xml (deflated 41%)
2022-12-01T10:52:34.5603296Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103959.xml (deflated 41%)
2022-12-01T10:52:34.5603881Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104002.xml (deflated 42%)
2022-12-01T10:52:34.5604459Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104003.xml (deflated 41%)
2022-12-01T10:52:34.5605077Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104005.xml (deflated 42%)
2022-12-01T10:52:34.5605669Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104010.xml (deflated 41%)
2022-12-01T10:52:34.5606254Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104016.xml (deflated 41%)
2022-12-01T10:52:34.5606830Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104019.xml (deflated 41%)
2022-12-01T10:52:34.5607412Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104022.xml (deflated 41%)
2022-12-01T10:52:34.5607994Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104026.xml (deflated 42%)
2022-12-01T10:52:34.5608580Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104027.xml (deflated 42%)
2022-12-01T10:52:34.5609152Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104029.xml (deflated 42%)
2022-12-01T10:52:34.5609780Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104030.xml (deflated 43%)
2022-12-01T10:52:34.5610365Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104031.xml (deflated 43%)
2022-12-01T10:52:34.5610950Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104033.xml (deflated 42%)
2022-12-01T10:52:34.5611522Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104034.xml (deflated 42%)
2022-12-01T10:52:34.5612196Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104035.xml (deflated 42%)
2022-12-01T10:52:34.5612782Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104037.xml (deflated 41%)
2022-12-01T10:52:34.5613372Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104040.xml (deflated 42%)
2022-12-01T10:52:34.5613944Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104043.xml (deflated 42%)
2022-12-01T10:52:34.5614532Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104044.xml (deflated 43%)
2022-12-01T10:52:34.5615117Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104046.xml (deflated 42%)
2022-12-01T10:52:34.5615701Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104047.xml (deflated 42%)
2022-12-01T10:52:34.5616361Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104051.xml (deflated 42%)
2022-12-01T10:52:34.5616937Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104054.xml (deflated 41%)
2022-12-01T10:52:34.5617519Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104057.xml (deflated 42%)
2022-12-01T10:52:34.5618105Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104059.xml (deflated 42%)
2022-12-01T10:52:34.5618693Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104101.xml (deflated 41%)
2022-12-01T10:52:34.5619265Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104104.xml (deflated 41%)
2022-12-01T10:52:34.5619904Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104107.xml (deflated 42%)
2022-12-01T10:52:34.5620588Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104109.xml (deflated 41%)
2022-12-01T10:52:34.5621174Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104112.xml (deflated 41%)
2022-12-01T10:52:34.5621745Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104116.xml (deflated 41%)
2022-12-01T10:52:34.5622328Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104119.xml (deflated 41%)
2022-12-01T10:52:34.5622914Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104122.xml (deflated 41%)
2022-12-01T10:52:34.5623500Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104126.xml (deflated 41%)
2022-12-01T10:52:34.5624156Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104129.xml (deflated 41%)
2022-12-01T10:52:34.5624747Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104133.xml (deflated 41%)
2022-12-01T10:52:34.5625331Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104136.xml (deflated 41%)
2022-12-01T10:52:34.5625918Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104140.xml (deflated 42%)
2022-12-01T10:52:34.5626491Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104143.xml (deflated 43%)
2022-12-01T10:52:34.5627080Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104144.xml (deflated 41%)
2022-12-01T10:52:34.5627666Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104148.xml (deflated 44%)
2022-12-01T10:52:34.5628253Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104149.xml (deflated 44%)
2022-12-01T10:52:34.5628822Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104151.xml (deflated 41%)
2022-12-01T10:52:34.5629408Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104154.xml (deflated 43%)
2022-12-01T10:52:34.5629996Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104155.xml (deflated 43%)
2022-12-01T10:52:34.5630578Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104157.xml (deflated 41%)
2022-12-01T10:52:34.5631149Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104200.xml (deflated 42%)
2022-12-01T10:52:34.5631731Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104203.xml (deflated 41%)
2022-12-01T10:52:34.5632315Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104207.xml (deflated 41%)
2022-12-01T10:52:34.5632899Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104210.xml (deflated 42%)
2022-12-01T10:52:34.5633481Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104211.xml (deflated 42%)
2022-12-01T10:52:34.5634059Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104213.xml (deflated 41%)
2022-12-01T10:52:34.5634687Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104216.xml (deflated 41%)
2022-12-01T10:52:34.5635280Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104220.xml (deflated 41%)
2022-12-01T10:52:34.5635864Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104223.xml (deflated 41%)
2022-12-01T10:52:34.5636433Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104226.xml (deflated 41%)
2022-12-01T10:52:34.5637019Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104230.xml (deflated 41%)
2022-12-01T10:52:34.5637604Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104233.xml (deflated 41%)
2022-12-01T10:52:34.5638191Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104237.xml (deflated 41%)
2022-12-01T10:52:34.5638802Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104240.xml (deflated 42%)
2022-12-01T10:52:34.5639386Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104241.xml (deflated 42%)
2022-12-01T10:52:34.5639972Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104243.xml (deflated 42%)
2022-12-01T10:52:34.5640562Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104244.xml (deflated 41%)
2022-12-01T10:52:34.5641135Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104247.xml (deflated 41%)
2022-12-01T10:52:34.5641723Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104250.xml (deflated 41%)
2022-12-01T10:52:34.5642309Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104254.xml (deflated 41%)
2022-12-01T10:52:34.5642892Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104257.xml (deflated 41%)
2022-12-01T10:52:34.5643468Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104300.xml (deflated 42%)
2022-12-01T10:52:34.5644051Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104304.xml (deflated 42%)
2022-12-01T10:52:34.5644637Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104307.xml (deflated 40%)
2022-12-01T10:52:34.5645223Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104310.xml (deflated 42%)
2022-12-01T10:52:34.5645798Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104314.xml (deflated 42%)
2022-12-01T10:52:34.5646378Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104317.xml (deflated 41%)
2022-12-01T10:52:34.5646963Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104318.xml (deflated 41%)
2022-12-01T10:52:34.5647600Z   adding: test/test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221201105058.xml (deflated 42%)
2022-12-01T10:52:34.5648262Z   adding: test/test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221201105100.xml (deflated 42%)
2022-12-01T10:52:34.5648968Z   adding: test/test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221201105103.xml (deflated 43%)
2022-12-01T10:52:34.5649648Z   adding: test/test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221201105104.xml (deflated 43%)
2022-12-01T10:52:34.5650310Z   adding: test/test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221201105105.xml (deflated 42%)
2022-12-01T10:52:34.5650964Z   adding: test/test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221201105107.xml (deflated 43%)
2022-12-01T10:52:34.5651632Z   adding: test/test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221201105109.xml (deflated 42%)
2022-12-01T10:52:34.5652379Z   adding: test/test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221201105112.xml (deflated 42%)
2022-12-01T10:52:34.5653048Z   adding: test/test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221201105115.xml (deflated 43%)
2022-12-01T10:52:34.5653761Z   adding: test/test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221201105116.xml (deflated 42%)
2022-12-01T10:52:34.5654428Z   adding: test/test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221201105117.xml (deflated 42%)
2022-12-01T10:52:34.5655092Z   adding: test/test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221201105118.xml (deflated 43%)
2022-12-01T10:52:34.5673837Z ##[group]Run # Remove any previous test reports if they exist
2022-12-01T10:52:34.5674154Z # Remove any previous test reports if they exist
2022-12-01T10:52:34.5674389Z rm -f usage-log-*.zip
2022-12-01T10:52:34.5674935Z # this workflow is also run in bazel build test, but we dont generate usage reports for it
2022-12-01T10:52:34.5675229Z # so check to see if the file exists first
2022-12-01T10:52:34.5675452Z if [ -f 'usage_log.txt' ]; then
2022-12-01T10:52:34.5675698Z     zip "usage-log-${FILE_SUFFIX}.zip" 'usage_log.txt'
2022-12-01T10:52:34.5675917Z fi
2022-12-01T10:52:34.5687238Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-12-01T10:52:34.5687460Z env:
2022-12-01T10:52:34.5687638Z   GIT_DEFAULT_BRANCH: master
2022-12-01T10:52:34.5687884Z   FILE_SUFFIX: test-distributed-2-2-linux.2xlarge_9818326018
2022-12-01T10:52:34.5688121Z ##[endgroup]
2022-12-01T10:52:34.6065125Z   adding: usage_log.txt (deflated 95%)
2022-12-01T10:52:34.6126387Z ##[group]Run seemethere/upload-artifact-s3@v5
2022-12-01T10:52:34.6126605Z with:
2022-12-01T10:52:34.6126797Z   s3-prefix: pytorch/pytorch/3591403534/1/artifact

2022-12-01T10:52:34.6127014Z   retention-days: 14
2022-12-01T10:52:34.6127210Z   if-no-files-found: warn
2022-12-01T10:52:34.6127404Z   path: test-jsons-*.zip
2022-12-01T10:52:34.6127588Z   name: artifact
2022-12-01T10:52:34.6127777Z   s3-bucket: gha-artifacts
2022-12-01T10:52:34.6127966Z   region: us-east-1
2022-12-01T10:52:34.6128118Z env:
2022-12-01T10:52:34.6128284Z   GIT_DEFAULT_BRANCH: master
2022-12-01T10:52:34.6128466Z ##[endgroup]
2022-12-01T10:52:34.9515079Z NOTE: s3-prefix specified, ignoring name parameter
2022-12-01T10:52:34.9515537Z With the provided path, there will be 1 file uploaded
2022-12-01T10:52:34.9516041Z Uploading to s3 prefix: pytorch/pytorch/3591403534/1/artifact
2022-12-01T10:52:34.9522770Z Starting upload of test-jsons-test-distributed-2-2-linux.2xlarge_9818326018.zip
2022-12-01T10:52:35.0950515Z Finished upload of test-jsons-test-distributed-2-2-linux.2xlarge_9818326018.zip
2022-12-01T10:52:35.1076014Z ##[group]Run seemethere/upload-artifact-s3@v5
2022-12-01T10:52:35.1076236Z with:
2022-12-01T10:52:35.1076428Z   s3-prefix: pytorch/pytorch/3591403534/1/artifact

2022-12-01T10:52:35.1076643Z   retention-days: 14
2022-12-01T10:52:35.1076936Z   if-no-files-found: error
2022-12-01T10:52:35.1077144Z   path: test-reports-*.zip
2022-12-01T10:52:35.1077330Z   name: artifact
2022-12-01T10:52:35.1077511Z   s3-bucket: gha-artifacts
2022-12-01T10:52:35.1077691Z   region: us-east-1
2022-12-01T10:52:35.1077854Z env:
2022-12-01T10:52:35.1078029Z   GIT_DEFAULT_BRANCH: master
2022-12-01T10:52:35.1078198Z ##[endgroup]
2022-12-01T10:52:35.4490689Z NOTE: s3-prefix specified, ignoring name parameter
2022-12-01T10:52:35.4491039Z With the provided path, there will be 1 file uploaded
2022-12-01T10:52:35.4491313Z Uploading to s3 prefix: pytorch/pytorch/3591403534/1/artifact
2022-12-01T10:52:35.4498174Z Starting upload of test-reports-test-distributed-2-2-linux.2xlarge_9818326018.zip
2022-12-01T10:52:35.6368116Z Finished upload of test-reports-test-distributed-2-2-linux.2xlarge_9818326018.zip
2022-12-01T10:52:35.6489060Z ##[group]Run seemethere/upload-artifact-s3@v5
2022-12-01T10:52:35.6489319Z with:
2022-12-01T10:52:35.6489520Z   s3-prefix: pytorch/pytorch/3591403534/1/artifact

2022-12-01T10:52:35.6489742Z   retention-days: 14
2022-12-01T10:52:35.6490012Z   if-no-files-found: ignore
2022-12-01T10:52:35.6490201Z   path: usage-log-*.zip
2022-12-01T10:52:35.6490385Z   name: artifact
2022-12-01T10:52:35.6490571Z   s3-bucket: gha-artifacts
2022-12-01T10:52:35.6490752Z   region: us-east-1
2022-12-01T10:52:35.6490924Z env:
2022-12-01T10:52:35.6491096Z   GIT_DEFAULT_BRANCH: master
2022-12-01T10:52:35.6491266Z ##[endgroup]
2022-12-01T10:52:35.9869353Z NOTE: s3-prefix specified, ignoring name parameter
2022-12-01T10:52:35.9869983Z With the provided path, there will be 1 file uploaded
2022-12-01T10:52:35.9870304Z Uploading to s3 prefix: pytorch/pytorch/3591403534/1/artifact
2022-12-01T10:52:35.9876986Z Starting upload of usage-log-test-distributed-2-2-linux.2xlarge_9818326018.zip
2022-12-01T10:52:36.1787266Z Finished upload of usage-log-test-distributed-2-2-linux.2xlarge_9818326018.zip
2022-12-01T10:52:36.1915284Z ##[group]Run set -x
2022-12-01T10:52:36.1915494Z set -x
2022-12-01T10:52:36.1915702Z python3 -m pip install -r requirements.txt
2022-12-01T10:52:36.1915957Z python3 -m pip install boto3==1.19.12
2022-12-01T10:52:36.1916250Z python3 -m tools.stats.print_test_stats --upload-to-s3 --compare-with-s3 test
2022-12-01T10:52:36.1927498Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-12-01T10:52:36.1927705Z env:
2022-12-01T10:52:36.1927885Z   GIT_DEFAULT_BRANCH: master
2022-12-01T10:52:36.1928088Z   AWS_DEFAULT_REGION: us-east-1
2022-12-01T10:52:36.1928268Z   BRANCH: pull/89997
2022-12-01T10:52:36.1928456Z   TEST_CONFIG: distributed
2022-12-01T10:52:36.1928639Z   SHARD_NUMBER: 2
2022-12-01T10:52:36.1928842Z   BUILD_ENVIRONMENT: linux-focal-py3.7-gcc7
2022-12-01T10:52:36.1929057Z   PR_NUMBER: 89997
2022-12-01T10:52:36.1929246Z   PYTORCH_RETRY_TEST_CASES: 1
2022-12-01T10:52:36.1929441Z   PYTORCH_OVERRIDE_FLAKY_SIGNAL: 1
2022-12-01T10:52:36.1929709Z   SHA1: c13d400bffe90e16b96520bbc8a41a6f0c9cd584
2022-12-01T10:52:36.1929913Z   TAG: 
2022-12-01T10:52:36.1930065Z   WORKFLOW_ID: 3591403534
2022-12-01T10:52:36.1930387Z   GITHUB_TOKEN: ***
2022-12-01T10:52:36.1930581Z   GHA_WORKFLOW_JOB_ID: 9818326018
2022-12-01T10:52:36.1930757Z ##[endgroup]
2022-12-01T10:52:36.1955944Z + python3 -m pip install -r requirements.txt
2022-12-01T10:52:36.4095971Z Defaulting to user installation because normal site-packages is not writeable
2022-12-01T10:52:36.4361926Z Requirement already satisfied: astunparse in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 2)) (1.6.3)
2022-12-01T10:52:36.4387722Z Requirement already satisfied: expecttest in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 3)) (0.1.4)
2022-12-01T10:52:36.4395688Z Requirement already satisfied: future in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 4)) (0.18.2)
2022-12-01T10:52:36.4405183Z Requirement already satisfied: hypothesis in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 5)) (6.58.2)
2022-12-01T10:52:36.4771421Z Requirement already satisfied: numpy in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 6)) (1.21.6)
2022-12-01T10:52:36.4780703Z Requirement already satisfied: psutil in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 7)) (5.9.1)
2022-12-01T10:52:36.4860631Z Requirement already satisfied: pyyaml in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 8)) (6.0)
2022-12-01T10:52:36.4868578Z Requirement already satisfied: requests in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 9)) (2.26.0)
2022-12-01T10:52:36.5092401Z Requirement already satisfied: setuptools in /usr/lib/python3.7/site-packages (from -r requirements.txt (line 10)) (49.1.3)
2022-12-01T10:52:36.5260958Z Requirement already satisfied: six in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 11)) (1.16.0)
2022-12-01T10:52:36.5269949Z Requirement already satisfied: types-dataclasses in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 12)) (0.6.6)
2022-12-01T10:52:36.5275366Z Requirement already satisfied: typing_extensions in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 13)) (4.4.0)
2022-12-01T10:52:36.5285144Z Requirement already satisfied: sympy in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 14)) (1.10.1)
2022-12-01T10:52:36.5306129Z Requirement already satisfied: wheel<1.0,>=0.23.0 in /home/ec2-user/.local/lib/python3.7/site-packages (from astunparse->-r requirements.txt (line 2)) (0.38.4)
2022-12-01T10:52:36.5321965Z Requirement already satisfied: exceptiongroup>=1.0.0; python_version < "3.11" in /home/ec2-user/.local/lib/python3.7/site-packages (from hypothesis->-r requirements.txt (line 5)) (1.0.4)
2022-12-01T10:52:36.5338433Z Requirement already satisfied: attrs>=19.2.0 in /home/ec2-user/.local/lib/python3.7/site-packages (from hypothesis->-r requirements.txt (line 5)) (22.1.0)
2022-12-01T10:52:36.5589903Z Requirement already satisfied: sortedcontainers<3.0.0,>=2.1.0 in /home/ec2-user/.local/lib/python3.7/site-packages (from hypothesis->-r requirements.txt (line 5)) (2.4.0)
2022-12-01T10:52:36.5600411Z Requirement already satisfied: idna<4,>=2.5; python_version >= "3" in /home/ec2-user/.local/lib/python3.7/site-packages (from requests->-r requirements.txt (line 9)) (3.4)
2022-12-01T10:52:36.5611561Z Requirement already satisfied: certifi>=2017.4.17 in /home/ec2-user/.local/lib/python3.7/site-packages (from requests->-r requirements.txt (line 9)) (2022.9.24)
2022-12-01T10:52:36.5619820Z Requirement already satisfied: urllib3<1.27,>=1.21.1 in /home/ec2-user/.local/lib/python3.7/site-packages (from requests->-r requirements.txt (line 9)) (1.26.13)
2022-12-01T10:52:36.5772411Z Requirement already satisfied: charset-normalizer~=2.0.0; python_version >= "3" in /home/ec2-user/.local/lib/python3.7/site-packages (from requests->-r requirements.txt (line 9)) (2.0.12)
2022-12-01T10:52:36.5791317Z Requirement already satisfied: mpmath>=0.19 in /home/ec2-user/.local/lib/python3.7/site-packages (from sympy->-r requirements.txt (line 14)) (1.2.1)
2022-12-01T10:52:36.6700280Z + python3 -m pip install boto3==1.19.12
2022-12-01T10:52:36.8819578Z Defaulting to user installation because normal site-packages is not writeable
2022-12-01T10:52:36.9005254Z Requirement already satisfied: boto3==1.19.12 in /home/ec2-user/.local/lib/python3.7/site-packages (1.19.12)
2022-12-01T10:52:36.9054485Z Requirement already satisfied: jmespath<1.0.0,>=0.7.1 in /home/ec2-user/.local/lib/python3.7/site-packages (from boto3==1.19.12) (0.10.0)
2022-12-01T10:52:36.9067152Z Requirement already satisfied: botocore<1.23.0,>=1.22.12 in /home/ec2-user/.local/lib/python3.7/site-packages (from boto3==1.19.12) (1.22.12)
2022-12-01T10:52:36.9108451Z Requirement already satisfied: s3transfer<0.6.0,>=0.5.0 in /home/ec2-user/.local/lib/python3.7/site-packages (from boto3==1.19.12) (0.5.2)
2022-12-01T10:52:36.9143988Z Requirement already satisfied: python-dateutil<3.0.0,>=2.1 in /home/ec2-user/.local/lib/python3.7/site-packages (from botocore<1.23.0,>=1.22.12->boto3==1.19.12) (2.8.2)
2022-12-01T10:52:36.9163613Z Requirement already satisfied: urllib3<1.27,>=1.25.4 in /home/ec2-user/.local/lib/python3.7/site-packages (from botocore<1.23.0,>=1.22.12->boto3==1.19.12) (1.26.13)
2022-12-01T10:52:36.9316336Z Requirement already satisfied: six>=1.5 in /home/ec2-user/.local/lib/python3.7/site-packages (from python-dateutil<3.0.0,>=2.1->botocore<1.23.0,>=1.22.12->boto3==1.19.12) (1.16.0)
2022-12-01T10:52:37.1216945Z + python3 -m tools.stats.print_test_stats --upload-to-s3 --compare-with-s3 test
2022-12-01T10:52:43.7859644Z [scribe] Scribe access token not provided, sending report via boto3...
2022-12-01T10:52:43.7860012Z 
2022-12-01T10:52:43.7860630Z ----- Historic stats comparison result ------
2022-12-01T10:52:43.7860936Z 
2022-12-01T10:52:43.7861196Z     job: linux-focal-py3.7-gcc7
2022-12-01T10:52:43.7861632Z     commit: c13d400bffe90e16b96520bbc8a41a6f0c9cd584
2022-12-01T10:52:43.7862182Z 
2022-12-01T10:52:43.7862454Z Commit graph (base is most recent master ancestor with at least one S3 report):
2022-12-01T10:52:43.7862778Z 
2022-12-01T10:52:43.7862901Z     : (master)
2022-12-01T10:52:43.7863196Z     |
2022-12-01T10:52:43.7863538Z     | * c13d400bff (HEAD)            total time  1213.33s
2022-12-01T10:52:43.7863890Z     | |
2022-12-01T10:52:43.7864200Z     | : (59 commits)
2022-12-01T10:52:43.7864498Z     |/
2022-12-01T10:52:43.7865339Z     * 67eb2d5952 (base)  12 reports, total time  2500.30s ± 1654.93s
2022-12-01T10:52:43.7865905Z     * 1c5ca724f4          6 reports, total time  2466.99s ± 1724.34s
2022-12-01T10:52:43.7866450Z     * 9d6109c4b0          6 reports, total time  2484.54s ± 1728.75s
2022-12-01T10:52:43.7867211Z     * 736adc0808          6 reports, total time  2481.20s ± 1744.38s
2022-12-01T10:52:43.7867760Z     * a348975e00          6 reports, total time  2497.04s ± 1746.31s
2022-12-01T10:52:43.7868304Z     * db13049b88          6 reports, total time  2514.30s ± 1779.23s
2022-12-01T10:52:43.7868863Z     * d07b85393a          6 reports, total time  2478.62s ± 1754.66s
2022-12-01T10:52:43.7869417Z     * ac25c210e5          6 reports, total time  2568.96s ± 1821.46s
2022-12-01T10:52:43.7869963Z     * 2355b6256b          6 reports, total time  2536.34s ± 1800.59s
2022-12-01T10:52:43.7872070Z     * 4f95f7ae9b          6 reports, total time  2504.50s ± 1759.28s
2022-12-01T10:52:43.7872476Z     |
2022-12-01T10:52:43.7872770Z     :
2022-12-01T10:52:43.7872951Z 
2022-12-01T10:52:43.7873159Z Removed  (across  826 suites)     0 tests, totaling     0.00s
2022-12-01T10:52:43.7873632Z Modified (across    0 suites)     0 tests, totaling     0.00s
2022-12-01T10:52:43.7874112Z Added    (across   82 suites)  1105 tests, totaling +1553.42s
2022-12-01T10:52:43.8478685Z ##[group]Run pytorch/test-infra/.github/actions/teardown-linux@main
2022-12-01T10:52:43.8478919Z with:
2022-12-01T10:52:43.8479078Z env:
2022-12-01T10:52:43.8479265Z   GIT_DEFAULT_BRANCH: master
2022-12-01T10:52:43.8479438Z ##[endgroup]
2022-12-01T10:52:43.8494486Z ##[group]Run set -eou pipefail
2022-12-01T10:52:43.8494711Z set -eou pipefail
2022-12-01T10:52:43.8494894Z 
2022-12-01T10:52:43.8495133Z echo "Holding runner for 2 hours until all ssh sessions have logged out"
2022-12-01T10:52:43.8495371Z for _ in $(seq 1440); do
2022-12-01T10:52:43.8495593Z     # Break if no ssh session exists anymore
2022-12-01T10:52:43.8495813Z     if [ "$(who)" = "" ]; then
2022-12-01T10:52:43.8495987Z       break
2022-12-01T10:52:43.8496160Z     fi
2022-12-01T10:52:43.8496333Z     echo "."
2022-12-01T10:52:43.8496503Z     sleep 5
2022-12-01T10:52:43.8496676Z done
2022-12-01T10:52:43.8508294Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-12-01T10:52:43.8508507Z env:
2022-12-01T10:52:43.8508689Z   GIT_DEFAULT_BRANCH: master
2022-12-01T10:52:43.8508881Z ##[endgroup]
2022-12-01T10:52:43.8533489Z Holding runner for 2 hours until all ssh sessions have logged out
2022-12-01T10:52:43.8571383Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty
2022-12-01T10:52:43.8571787Z # ignore expansion of "docker ps -q" since it could be empty
2022-12-01T10:52:43.8572041Z # shellcheck disable=SC2046
2022-12-01T10:52:43.8572268Z docker stop $(docker ps -q) || true
2022-12-01T10:52:43.8572484Z # Prune all of the docker images
2022-12-01T10:52:43.8572702Z docker system prune -af
2022-12-01T10:52:43.8583973Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-12-01T10:52:43.8584181Z env:
2022-12-01T10:52:43.8584361Z   GIT_DEFAULT_BRANCH: master
2022-12-01T10:52:43.8584553Z ##[endgroup]
2022-12-01T10:52:44.2321189Z 0bf5643a9780
2022-12-01T10:52:44.7767207Z Deleted Containers:
2022-12-01T10:52:44.7767575Z 0bf5643a9780fdbcf8441394ea970b88da0db91cdab4fb259a94a9efce50888d
2022-12-01T10:52:44.7767964Z 
2022-12-01T10:52:47.3328028Z Deleted Images:
2022-12-01T10:52:47.3329094Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-py3.7-gcc7:fa72f5a0a230eb632055220542038bd4ceca184b
2022-12-01T10:52:47.3330053Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-py3.7-gcc7@sha256:7d4daa1530ad815417dd92428a12643a4a8b0e93e820e2eb2d4e8818a4e57ba2
2022-12-01T10:52:47.3330494Z deleted: sha256:e8be7981a8238969daaef272b84b69737c5858187c9b10870da76dec899f6864
2022-12-01T10:52:47.3330804Z deleted: sha256:dd542825775450ac98a8f341e744c9b998cf9f91f0b678b696bdfdcc8197900b
2022-12-01T10:52:47.3331136Z deleted: sha256:c2f995bd392fa39048aba9ecdbe2ce96fb0da354f391973cab978f1fdc1bafe9
2022-12-01T10:52:47.3331503Z deleted: sha256:0ba22d6d32aea10bff37a734be2eabf0ca45365a8c84009c11c72697bbeb57ac
2022-12-01T10:52:47.3331908Z deleted: sha256:1641ef8553a9d06a751fe1c9fa58edae658b58be5e4452cf0e6c22fe7e9e9f7a
2022-12-01T10:52:47.3332241Z deleted: sha256:6fe08b57b8677fadf24df12c30c6b075caa4e95cff1e8121398e7712e7cc816b
2022-12-01T10:52:47.3332578Z deleted: sha256:b535a07f21c50706835dc1c6a9f6134e10f492c0a72ef3edd3d61eb8b057318a
2022-12-01T10:52:47.3332903Z deleted: sha256:ad51a4c1d9f796ae822ae9448a0439daec4dc5352ab73405d5170d8320d6569b
2022-12-01T10:52:47.3333227Z deleted: sha256:cc3572e58e1eadfed379b3421fc485a2fec838dbe2f03f9482a844387875539b
2022-12-01T10:52:47.3333541Z deleted: sha256:f31159505662112f25c640d83a5486c60dcea9996cfac1468ae7b74838077420
2022-12-01T10:52:47.3333854Z deleted: sha256:5d54ecd58fecab272c8614e2c443272f47c9245eaf4b5be502820db913ddd776
2022-12-01T10:52:47.3334183Z deleted: sha256:e8fcb42d7c24a5f6cd663f59d8fd82a2ad892e66046dc0d36ee4bdadf1282b52
2022-12-01T10:52:47.3334521Z deleted: sha256:934da62f4ce129178b2355504cdb61701bffc48d1bcf3b1ce0ca6e4c89bdfa31
2022-12-01T10:52:47.3335015Z deleted: sha256:8dccdf71b42bc4d3717c42c3d96812509d1a909bebd7848d0ddae3cc40170990
2022-12-01T10:52:47.3335358Z deleted: sha256:d7cb3c132d871b52eaf2533cfbc7ebca30405eff1b313986ce565b057fca56d8
2022-12-01T10:52:47.3335691Z deleted: sha256:a5fa381fecd94b52ee79c5c4ce30770402b16f7ee3dc77f7f1dccb386626b1ae
2022-12-01T10:52:47.3336066Z deleted: sha256:de8b14689d2a8b030e41a969f6faf30e9dee1048aac1c5d9a38a911b6af1ccca
2022-12-01T10:52:47.3336412Z deleted: sha256:ddcc5d19eb58848e898aedcc3a221bc40ebba2752ba3b7bd50babd76c1e225ae
2022-12-01T10:52:47.3336751Z deleted: sha256:a67b9c1632db819366358a5052f33ed59ff1e1cd03e0cc15e82d8c89a9417b97
2022-12-01T10:52:47.3337075Z deleted: sha256:a5fc4c6c0aaad2d5fcd0d4cc4cf194a8892115d9e7faf6c578e4826937a72d34
2022-12-01T10:52:47.3337395Z deleted: sha256:f33cf9181261127561bc276f483d4f9927a63395c4343028c589ac8905dd0cd7
2022-12-01T10:52:47.3337698Z deleted: sha256:4364081b1a5daf042c41fa0b07d65e0488c3a988dcea07cec40851d8bb481c01
2022-12-01T10:52:47.3338019Z deleted: sha256:2c4beb5d46c3bdc6af376c4a784f3ea2032feda5143dc9b56be143568b8d71cd
2022-12-01T10:52:47.3338337Z deleted: sha256:3e002fe51ea8377736f778797676c87d37546763f35bc5f14a32121200c634ae
2022-12-01T10:52:47.3338646Z deleted: sha256:d1e577d22ad4a397fc4186062a6a2fbb898c4fb7c325794f0760c084ac4e6831
2022-12-01T10:52:47.3338951Z deleted: sha256:0781305ea794027575e6910f20b6c90cc225056d0112fa7f36ac8008431f5791
2022-12-01T10:52:47.3339232Z deleted: sha256:79f50ef439b6855c9ca6d4be55467b692799f788388447976d8aab0a26d805f4
2022-12-01T10:52:47.3339545Z deleted: sha256:fcfdd0674d63c370d31701118409d5e8b4d9b5eaa36b410f032f5fccf29e4f11
2022-12-01T10:52:47.3339866Z deleted: sha256:c868e869e0efb64e8a908cf5a276657da1190183a5aa433a72e1fda6313e8841
2022-12-01T10:52:47.3340181Z deleted: sha256:0ac36be062e7b703a8ffa26b178cc9dd7c19c377bd781f6a369d50516bdca8cb
2022-12-01T10:52:47.3340662Z deleted: sha256:6c9628b9c75a72926209bea0ff44fb2efe306ecb21d57908bb514b1d81b5de8e
2022-12-01T10:52:47.3341001Z deleted: sha256:07c5d0507acce428b3637d5f713aed3fc6dd2dfad749e1961ec04f85d2a3e395
2022-12-01T10:52:47.3341335Z deleted: sha256:278ae5d4c6743b99236e1f2e5df4c346df5f068fa1bce0ba227861042971d612
2022-12-01T10:52:47.3341657Z deleted: sha256:d53c6cc3bfa4330c26ff3a73093bf35a7f28e7fc824782cfc1e00fcbc0390574
2022-12-01T10:52:47.3342086Z deleted: sha256:0a926d106d3065ecc114c8690aee4bab27d4f64ee81c96f90c4daf7a3c03fe7f
2022-12-01T10:52:47.3342413Z deleted: sha256:ab29088e02867cf532a2e8cfa0e1b0a1d13b428459232c7b1a4bfe961fd5a511
2022-12-01T10:52:47.3342844Z deleted: sha256:985cef3ba9e1ea2ed2249284a7478858f8463282738122494ab18c65fbbb7c7d
2022-12-01T10:52:47.3343316Z deleted: sha256:67ca28cabbfd7a80760061b52dbcfe8f5401b867526206d76c710a11cf9ab38e
2022-12-01T10:52:47.3343650Z deleted: sha256:2c9d0b64a3450004672e73e7ace7bc49da29654fd1d603ecf96e9161708726a3
2022-12-01T10:52:47.3344026Z deleted: sha256:efb1cb2fb03e227ccaa27426fae8f80bf1c082d14632685fff079b0c20111d3b
2022-12-01T10:52:47.3344461Z deleted: sha256:2bc3f10f3904b8fff87213a8a2325ab12e4048c0a19247b3d82886066b37a60d
2022-12-01T10:52:47.3344788Z deleted: sha256:adf0c04002f67cf46889421aed44a0e23618d009f3bbfb6aa94e3fced3cdb131
2022-12-01T10:52:47.3345103Z deleted: sha256:5b43511c57d66e58ff42011707835b8cc6971ed316454cfd950e0a5a12138631
2022-12-01T10:52:47.3345408Z deleted: sha256:ad9d96076f7f0bce4820b7828780dbf5328c1c45b9b6c93cfc21fa6c78e66765
2022-12-01T10:52:47.3345724Z deleted: sha256:545999a778d60ca8c6f4e76973a1c707cc626147f1e4d83eebbad6a57101cce0
2022-12-01T10:52:47.3346034Z deleted: sha256:cfcad11f69b44005150106d3108821c82a41765afbeb809208c1e0f49d5b6135
2022-12-01T10:52:47.3346371Z deleted: sha256:f7c8d5bcafe4ebe33f704b703a618803e7ade038c4cef7b4c3b520e9a100cd53
2022-12-01T10:52:47.3346699Z deleted: sha256:4bad106558a8cdd7564af2e1486b545ff1018677d73a8dcb029ecea2ad5afa9a
2022-12-01T10:52:47.3347017Z deleted: sha256:b01943462df81aa35c0f222112f8a7a13b09ec19be40907c2a81fded5e894e14
2022-12-01T10:52:47.3347344Z deleted: sha256:74a161fbdc63f2caa195797d69cca089d2f2ec781d7e7dc78fac88f57c610741
2022-12-01T10:52:47.3347767Z deleted: sha256:d94fdd2bb4227aed66703d740136d3bba2d45ac1c65930624fc2e7ab50021822
2022-12-01T10:52:47.3348096Z deleted: sha256:b348cd6909427063b9b36464ac9e6edb7d42ade2f164c5a98ee3ec37b1074e72
2022-12-01T10:52:47.3348420Z deleted: sha256:0fba67987a5ab74fca03bb2b1d623325a4c40e0e42a74379322b082612b8b445
2022-12-01T10:52:47.3348727Z deleted: sha256:030e74143974068ec1ab10e03d50f82455d36264aa70c98bd0fad278e4425d28
2022-12-01T10:52:47.3349032Z deleted: sha256:b7fcb4ae2181c6c159bcc2f4c7b624150a8245dbdeee9d8601d008b5cd8be8fd
2022-12-01T10:52:47.3349351Z deleted: sha256:4472f94e146cc542a9111182b122e9a9c93a8f7022a48ce3c94b600be8fcdfaa
2022-12-01T10:52:47.3349671Z deleted: sha256:33bc8c1ba43ca724221fd1a8c992a420b1d8de4c85372defdc409e5c88c005b9
2022-12-01T10:52:47.3349990Z deleted: sha256:7070a4ad8ac000d6910f8bab3e00671f9ea4e7094064da08eb1892332166b92d
2022-12-01T10:52:47.3350303Z deleted: sha256:b40ed86654e59e1012e1716d5384910f8c3bb58274b7b00fca564a53e9897ba3
2022-12-01T10:52:47.3350474Z 
2022-12-01T10:52:47.3350582Z Total reclaimed space: 6.71GB
2022-12-01T10:52:47.3392503Z Post job cleanup.
2022-12-01T10:52:47.3422599Z Post job cleanup.
2022-12-01T10:52:47.4427553Z [command]/usr/bin/git version
2022-12-01T10:52:47.4470129Z git version 2.37.1
2022-12-01T10:52:47.4510760Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/4e35e68e-043e-4b48-a2ca-5378ad733d82' before making global git config changes
2022-12-01T10:52:47.4511587Z Adding repository directory to the temporary git global config as a safe directory
2022-12-01T10:52:47.4516356Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch
2022-12-01T10:52:47.4553900Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand
2022-12-01T10:52:47.4585404Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :
2022-12-01T10:52:47.4848350Z Entering 'android/libs/fbjni'
2022-12-01T10:52:47.4882981Z Entering 'third_party/FP16'
2022-12-01T10:52:47.4919590Z Entering 'third_party/FXdiv'
2022-12-01T10:52:47.4954079Z Entering 'third_party/NNPACK'
2022-12-01T10:52:47.4989727Z Entering 'third_party/QNNPACK'
2022-12-01T10:52:47.5024357Z Entering 'third_party/VulkanMemoryAllocator'
2022-12-01T10:52:47.5058444Z Entering 'third_party/XNNPACK'
2022-12-01T10:52:47.5101636Z Entering 'third_party/benchmark'
2022-12-01T10:52:47.5134730Z Entering 'third_party/cpuinfo'
2022-12-01T10:52:47.5169036Z Entering 'third_party/cub'
2022-12-01T10:52:47.5202746Z Entering 'third_party/cudnn_frontend'
2022-12-01T10:52:47.5241434Z Entering 'third_party/cutlass'
2022-12-01T10:52:47.5282554Z Entering 'third_party/eigen'
2022-12-01T10:52:47.5319177Z Entering 'third_party/fbgemm'
2022-12-01T10:52:47.5353471Z Entering 'third_party/fbgemm/third_party/asmjit'
2022-12-01T10:52:47.5386953Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2022-12-01T10:52:47.5421583Z Entering 'third_party/fbgemm/third_party/googletest'
2022-12-01T10:52:47.5456775Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2022-12-01T10:52:47.5492312Z Entering 'third_party/flatbuffers'
2022-12-01T10:52:47.5529574Z Entering 'third_party/fmt'
2022-12-01T10:52:47.5563487Z Entering 'third_party/foxi'
2022-12-01T10:52:47.5598369Z Entering 'third_party/gemmlowp/gemmlowp'
2022-12-01T10:52:47.5631820Z Entering 'third_party/gloo'
2022-12-01T10:52:47.5666139Z Entering 'third_party/googletest'
2022-12-01T10:52:47.5700596Z Entering 'third_party/ideep'
2022-12-01T10:52:47.5735048Z Entering 'third_party/ideep/mkl-dnn'
2022-12-01T10:52:47.5770336Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN'
2022-12-01T10:52:47.5811264Z Entering 'third_party/ios-cmake'
2022-12-01T10:52:47.5846161Z Entering 'third_party/ittapi'
2022-12-01T10:52:47.5880595Z Entering 'third_party/kineto'
2022-12-01T10:52:47.5914606Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2022-12-01T10:52:47.5949270Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2022-12-01T10:52:47.5984110Z Entering 'third_party/nccl/nccl'
2022-12-01T10:52:47.6018238Z Entering 'third_party/neon2sse'
2022-12-01T10:52:47.6052290Z Entering 'third_party/nlohmann'
2022-12-01T10:52:47.6087616Z Entering 'third_party/onnx'
2022-12-01T10:52:47.6134222Z Entering 'third_party/onnx/third_party/benchmark'
2022-12-01T10:52:47.6168713Z Entering 'third_party/onnx/third_party/pybind11'
2022-12-01T10:52:47.6204832Z Entering 'third_party/onnx-tensorrt'
2022-12-01T10:52:47.6239524Z Entering 'third_party/onnx-tensorrt/third_party/onnx'
2022-12-01T10:52:47.6277325Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'
2022-12-01T10:52:47.6311626Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'
2022-12-01T10:52:47.6346401Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'
2022-12-01T10:52:47.6386473Z Entering 'third_party/pocketfft'
2022-12-01T10:52:47.6421204Z Entering 'third_party/protobuf'
2022-12-01T10:52:47.6459066Z Entering 'third_party/protobuf/third_party/benchmark'
2022-12-01T10:52:47.6493104Z Entering 'third_party/protobuf/third_party/googletest'
2022-12-01T10:52:47.6528895Z Entering 'third_party/psimd'
2022-12-01T10:52:47.6563473Z Entering 'third_party/pthreadpool'
2022-12-01T10:52:47.6597155Z Entering 'third_party/pybind11'
2022-12-01T10:52:47.6632251Z Entering 'third_party/python-enum'
2022-12-01T10:52:47.6665674Z Entering 'third_party/python-peachpy'
2022-12-01T10:52:47.6698726Z Entering 'third_party/python-six'
2022-12-01T10:52:47.6732592Z Entering 'third_party/sleef'
2022-12-01T10:52:47.6766922Z Entering 'third_party/tbb'
2022-12-01T10:52:47.6802721Z Entering 'third_party/tensorpipe'
2022-12-01T10:52:47.6837399Z Entering 'third_party/tensorpipe/third_party/googletest'
2022-12-01T10:52:47.6870390Z Entering 'third_party/tensorpipe/third_party/libnop'
2022-12-01T10:52:47.6902991Z Entering 'third_party/tensorpipe/third_party/libuv'
2022-12-01T10:52:47.6936098Z Entering 'third_party/tensorpipe/third_party/pybind11'
2022-12-01T10:52:47.6971144Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2022-12-01T10:52:47.7006950Z Entering 'third_party/zstd'
2022-12-01T10:52:47.7054849Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader
2022-12-01T10:52:47.7081185Z http.https://github.com/.extraheader
2022-12-01T10:52:47.7089047Z [command]/usr/bin/git config --local --unset-all http.https://github.com/.extraheader
2022-12-01T10:52:47.7122265Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :
2022-12-01T10:52:47.7384174Z Entering 'android/libs/fbjni'
2022-12-01T10:52:47.7404048Z http.https://github.com/.extraheader
2022-12-01T10:52:47.7431680Z Entering 'third_party/FP16'
2022-12-01T10:52:47.7452749Z http.https://github.com/.extraheader
2022-12-01T10:52:47.7479469Z Entering 'third_party/FXdiv'
2022-12-01T10:52:47.7500603Z http.https://github.com/.extraheader
2022-12-01T10:52:47.7525711Z Entering 'third_party/NNPACK'
2022-12-01T10:52:47.7546917Z http.https://github.com/.extraheader
2022-12-01T10:52:47.7572934Z Entering 'third_party/QNNPACK'
2022-12-01T10:52:47.7592956Z http.https://github.com/.extraheader
2022-12-01T10:52:47.7619394Z Entering 'third_party/VulkanMemoryAllocator'
2022-12-01T10:52:47.7640456Z http.https://github.com/.extraheader
2022-12-01T10:52:47.7666906Z Entering 'third_party/XNNPACK'
2022-12-01T10:52:47.7686739Z http.https://github.com/.extraheader
2022-12-01T10:52:47.7723613Z Entering 'third_party/benchmark'
2022-12-01T10:52:47.7743853Z http.https://github.com/.extraheader
2022-12-01T10:52:47.7769601Z Entering 'third_party/cpuinfo'
2022-12-01T10:52:47.7789308Z http.https://github.com/.extraheader
2022-12-01T10:52:47.7815951Z Entering 'third_party/cub'
2022-12-01T10:52:47.7836380Z http.https://github.com/.extraheader
2022-12-01T10:52:47.7863452Z Entering 'third_party/cudnn_frontend'
2022-12-01T10:52:47.7883332Z http.https://github.com/.extraheader
2022-12-01T10:52:47.7915650Z Entering 'third_party/cutlass'
2022-12-01T10:52:47.7935643Z http.https://github.com/.extraheader
2022-12-01T10:52:47.7969600Z Entering 'third_party/eigen'
2022-12-01T10:52:47.7989281Z http.https://github.com/.extraheader
2022-12-01T10:52:47.8017720Z Entering 'third_party/fbgemm'
2022-12-01T10:52:47.8037754Z http.https://github.com/.extraheader
2022-12-01T10:52:47.8065360Z Entering 'third_party/fbgemm/third_party/asmjit'
2022-12-01T10:52:47.8085071Z http.https://github.com/.extraheader
2022-12-01T10:52:47.8111629Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2022-12-01T10:52:47.8131211Z http.https://github.com/.extraheader
2022-12-01T10:52:47.8157743Z Entering 'third_party/fbgemm/third_party/googletest'
2022-12-01T10:52:47.8176997Z http.https://github.com/.extraheader
2022-12-01T10:52:47.8204154Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2022-12-01T10:52:47.8224651Z http.https://github.com/.extraheader
2022-12-01T10:52:47.8252032Z Entering 'third_party/flatbuffers'
2022-12-01T10:52:47.8271795Z http.https://github.com/.extraheader
2022-12-01T10:52:47.8300557Z Entering 'third_party/fmt'
2022-12-01T10:52:47.8320747Z http.https://github.com/.extraheader
2022-12-01T10:52:47.8347715Z Entering 'third_party/foxi'
2022-12-01T10:52:47.8367076Z http.https://github.com/.extraheader
2022-12-01T10:52:47.8393269Z Entering 'third_party/gemmlowp/gemmlowp'
2022-12-01T10:52:47.8413291Z http.https://github.com/.extraheader
2022-12-01T10:52:47.8439336Z Entering 'third_party/gloo'
2022-12-01T10:52:47.8459001Z http.https://github.com/.extraheader
2022-12-01T10:52:47.8485779Z Entering 'third_party/googletest'
2022-12-01T10:52:47.8505618Z http.https://github.com/.extraheader
2022-12-01T10:52:47.8532110Z Entering 'third_party/ideep'
2022-12-01T10:52:47.8552060Z http.https://github.com/.extraheader
2022-12-01T10:52:47.8577625Z Entering 'third_party/ideep/mkl-dnn'
2022-12-01T10:52:47.8597265Z http.https://github.com/.extraheader
2022-12-01T10:52:47.8626039Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN'
2022-12-01T10:52:47.8645920Z http.https://github.com/.extraheader
2022-12-01T10:52:47.8679307Z Entering 'third_party/ios-cmake'
2022-12-01T10:52:47.8699976Z http.https://github.com/.extraheader
2022-12-01T10:52:47.8727149Z Entering 'third_party/ittapi'
2022-12-01T10:52:47.8747406Z http.https://github.com/.extraheader
2022-12-01T10:52:47.8773206Z Entering 'third_party/kineto'
2022-12-01T10:52:47.8795719Z http.https://github.com/.extraheader
2022-12-01T10:52:47.8824040Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2022-12-01T10:52:47.8843753Z http.https://github.com/.extraheader
2022-12-01T10:52:47.8873386Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2022-12-01T10:52:47.8893624Z http.https://github.com/.extraheader
2022-12-01T10:52:47.8920932Z Entering 'third_party/nccl/nccl'
2022-12-01T10:52:47.8941453Z http.https://github.com/.extraheader
2022-12-01T10:52:47.8968047Z Entering 'third_party/neon2sse'
2022-12-01T10:52:47.8987925Z http.https://github.com/.extraheader
2022-12-01T10:52:47.9014033Z Entering 'third_party/nlohmann'
2022-12-01T10:52:47.9033711Z http.https://github.com/.extraheader
2022-12-01T10:52:47.9061655Z Entering 'third_party/onnx'
2022-12-01T10:52:47.9084121Z http.https://github.com/.extraheader
2022-12-01T10:52:47.9123354Z Entering 'third_party/onnx/third_party/benchmark'
2022-12-01T10:52:47.9147060Z http.https://github.com/.extraheader
2022-12-01T10:52:47.9174447Z Entering 'third_party/onnx/third_party/pybind11'
2022-12-01T10:52:47.9194715Z http.https://github.com/.extraheader
2022-12-01T10:52:47.9226006Z Entering 'third_party/onnx-tensorrt'
2022-12-01T10:52:47.9246435Z http.https://github.com/.extraheader
2022-12-01T10:52:47.9272324Z Entering 'third_party/onnx-tensorrt/third_party/onnx'
2022-12-01T10:52:47.9292501Z http.https://github.com/.extraheader
2022-12-01T10:52:47.9323587Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'
2022-12-01T10:52:47.9346192Z http.https://github.com/.extraheader
2022-12-01T10:52:47.9372997Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'
2022-12-01T10:52:47.9394262Z http.https://github.com/.extraheader
2022-12-01T10:52:47.9423477Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'
2022-12-01T10:52:47.9442814Z http.https://github.com/.extraheader
2022-12-01T10:52:47.9474999Z Entering 'third_party/pocketfft'
2022-12-01T10:52:47.9495588Z http.https://github.com/.extraheader
2022-12-01T10:52:47.9521451Z Entering 'third_party/protobuf'
2022-12-01T10:52:47.9541995Z http.https://github.com/.extraheader
2022-12-01T10:52:47.9575542Z Entering 'third_party/protobuf/third_party/benchmark'
2022-12-01T10:52:47.9596834Z http.https://github.com/.extraheader
2022-12-01T10:52:47.9625926Z Entering 'third_party/protobuf/third_party/googletest'
2022-12-01T10:52:47.9646510Z http.https://github.com/.extraheader
2022-12-01T10:52:47.9675911Z Entering 'third_party/psimd'
2022-12-01T10:52:47.9697002Z http.https://github.com/.extraheader
2022-12-01T10:52:47.9723869Z Entering 'third_party/pthreadpool'
2022-12-01T10:52:47.9744363Z http.https://github.com/.extraheader
2022-12-01T10:52:47.9771105Z Entering 'third_party/pybind11'
2022-12-01T10:52:47.9792653Z http.https://github.com/.extraheader
2022-12-01T10:52:47.9855209Z Entering 'third_party/python-enum'
2022-12-01T10:52:47.9875641Z http.https://github.com/.extraheader
2022-12-01T10:52:47.9902143Z Entering 'third_party/python-peachpy'
2022-12-01T10:52:47.9922221Z http.https://github.com/.extraheader
2022-12-01T10:52:47.9949591Z Entering 'third_party/python-six'
2022-12-01T10:52:47.9969347Z http.https://github.com/.extraheader
2022-12-01T10:52:47.9996355Z Entering 'third_party/sleef'
2022-12-01T10:52:48.0016673Z http.https://github.com/.extraheader
2022-12-01T10:52:48.0044036Z Entering 'third_party/tbb'
2022-12-01T10:52:48.0065712Z http.https://github.com/.extraheader
2022-12-01T10:52:48.0118042Z Entering 'third_party/tensorpipe'
2022-12-01T10:52:48.0137696Z http.https://github.com/.extraheader
2022-12-01T10:52:48.0163977Z Entering 'third_party/tensorpipe/third_party/googletest'
2022-12-01T10:52:48.0184414Z http.https://github.com/.extraheader
2022-12-01T10:52:48.0210722Z Entering 'third_party/tensorpipe/third_party/libnop'
2022-12-01T10:52:48.0230348Z http.https://github.com/.extraheader
2022-12-01T10:52:48.0256377Z Entering 'third_party/tensorpipe/third_party/libuv'
2022-12-01T10:52:48.0276085Z http.https://github.com/.extraheader
2022-12-01T10:52:48.0305088Z Entering 'third_party/tensorpipe/third_party/pybind11'
2022-12-01T10:52:48.0324611Z http.https://github.com/.extraheader
2022-12-01T10:52:48.0350045Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2022-12-01T10:52:48.0369423Z http.https://github.com/.extraheader
2022-12-01T10:52:48.0397441Z Entering 'third_party/zstd'
2022-12-01T10:52:48.0417742Z http.https://github.com/.extraheader
2022-12-01T10:52:48.0655685Z Cleaning up orphan processes